2024-09-11T08:26:01.290 INFO:root:teuthology version: 1.2.2.dev2+g3752d38 2024-09-11T08:26:01.312 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T08:26:01.388 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659 branch: squid description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} email: ceph-qa@ceph.com first_in_suite: false job_id: '7900659' kernel: kdb: 1 sha1: distro ktype: distro last_in_suite: false machine_type: smithi meta: - desc: run two ceph clusters and install rbd-mirror - desc: configure the permissions for client.mirror - desc: 2 ceph clusters with 1 mon and 3 osds each - desc: run four rbd-mirror daemons per cluster - desc: run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon name: teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi no_nested_subset: false openstack: - volumes: count: 3 size: 30 os_type: ubuntu os_version: '22.04' overrides: admin_socket: branch: squid ceph: conf: client: debug rbd: 20 debug rbd_mirror: 15 log to stderr: false rbd default features: 125 rbd mirror image policy type: simple client.mirror.0: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.1: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.2: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid client.mirror.3: admin socket: /var/run/ceph/rbd-mirror.$cluster-$name.asok pid file: /var/run/ceph/rbd-mirror.$cluster-$name.pid global: mon client directed command retry: 5 mon warn on pool no app: false ms inject socket failures: 5000 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd memory target: 1610612736 osd objectstore: bluestore flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(OSD_SLOW_PING_TIME sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} install: ceph: flavor: default sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: squid sha1: daaf78de334647fc0c800f4ac495cc9475a26bae owner: scheduled_teuthology@teuthology priority: 100 repo: https://git.ceph.com/ceph.git roles: - - cluster1.mon.a - cluster1.mgr.x - cluster2.mgr.x - cluster1.osd.0 - cluster1.osd.1 - cluster1.osd.2 - cluster1.client.0 - cluster2.client.0 - - cluster2.mon.a - cluster2.osd.0 - cluster2.osd.1 - cluster2.osd.2 - cluster1.client.mirror - cluster1.client.mirror.0 - cluster1.client.mirror.1 - cluster1.client.mirror.2 - cluster1.client.mirror.3 - cluster1.client.mirror.4 - cluster1.client.mirror.5 - cluster1.client.mirror.6 - cluster2.client.mirror - cluster2.client.mirror.0 - cluster2.client.mirror.1 - cluster2.client.mirror.2 - cluster2.client.mirror.3 - cluster2.client.mirror.4 - cluster2.client.mirror.5 - cluster2.client.mirror.6 seed: 6175 sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc sleep_before_teardown: 0 subset: 88/128 suite: rbd suite_branch: squid suite_path: /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa suite_relpath: qa suite_repo: https://git.ceph.com/ceph.git suite_sha1: daaf78de334647fc0c800f4ac495cc9475a26bae targets: smithi073.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuh0QeHx3fryzEpQpK5xbTHvxwgQg32edZPQTXOSJGseSUdnXvrij6JDuffVEHtSPdwy3PgxAkZctvxRi6AXoo= smithi077.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZ2WuzHB7zjUUpe0P9YBQNHv5XrPIF9koGEMtfiHnIw/SlLQrufkVBxayqxTGeezl22EHKhHzsda1D5Jb1zis4= tasks: - install: extra_packages: - rbd-mirror - ceph: cluster: cluster1 - ceph: cluster: cluster2 - exec: cluster1.client.mirror.0: - sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd' - sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd' - rbd-mirror: client: cluster1.client.mirror.0 thrash: true - rbd-mirror: client: cluster1.client.mirror.1 thrash: true - rbd-mirror: client: cluster1.client.mirror.2 thrash: true - rbd-mirror: client: cluster1.client.mirror.3 thrash: true - rbd-mirror: client: cluster2.client.mirror.0 thrash: true - rbd-mirror: client: cluster2.client.mirror.1 thrash: true - rbd-mirror: client: cluster2.client.mirror.2 thrash: true - rbd-mirror: client: cluster2.client.mirror.3 thrash: true - rbd-mirror-thrash: cluster: cluster1 - rbd-mirror-thrash: cluster: cluster2 - workunit: clients: cluster1.client.mirror: - rbd/rbd_mirror_stress.sh env: CEPH_ARGS: '' MIRROR_POOL_MODE: image RBD_IMAGE_FEATURES: layering,exclusive-lock RBD_MIRROR_INSTANCES: '4' RBD_MIRROR_MODE: snapshot RBD_MIRROR_USE_EXISTING_CLUSTER: '1' RBD_MIRROR_USE_RBD_MIRROR: '1' timeout: 6h teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a timestamp: 2024-09-10_21:16:03 tube: smithi user: teuthology verbose: false worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856 2024-09-11T08:26:01.389 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa; will attempt to use it 2024-09-11T08:26:01.389 INFO:teuthology.run:Found tasks at /home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa/tasks 2024-09-11T08:26:01.390 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-09-11T08:26:01.391 INFO:teuthology.task.internal:Checking packages... 2024-09-11T08:26:01.414 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc' 2024-09-11T08:26:01.414 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-09-11T08:26:01.414 INFO:teuthology.packaging:ref: None 2024-09-11T08:26:01.414 INFO:teuthology.packaging:tag: None 2024-09-11T08:26:01.414 INFO:teuthology.packaging:branch: squid 2024-09-11T08:26:01.414 INFO:teuthology.packaging:sha1: b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T08:26:01.414 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&ref=squid 2024-09-11T08:26:01.648 INFO:teuthology.task.internal:Found packages for ceph version 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:26:01.650 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-09-11T08:26:01.656 INFO:teuthology.task.internal:no buildpackages task found 2024-09-11T08:26:01.656 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-09-11T08:26:01.665 INFO:teuthology.task.internal:Saving configuration 2024-09-11T08:26:01.683 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-09-11T08:26:01.689 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-09-11T08:26:01.715 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi073.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 08:20:29.708461', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIKFkOILmuMFJCCvy/ApZApp3aMlkqWidT6wuukBmEHpx'} 2024-09-11T08:26:01.736 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi077.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '22.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-09-11 08:20:29.706893', 'locked_by': 'scheduled_teuthology@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIK+u8ESLiQcMlQfrYuhZMEDxZvxHl59Rc/JjOWob8M73'} 2024-09-11T08:26:01.736 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-09-11T08:26:01.743 INFO:teuthology.task.internal:roles: ubuntu@smithi073.front.sepia.ceph.com - ['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'] 2024-09-11T08:26:01.744 INFO:teuthology.task.internal:roles: ubuntu@smithi077.front.sepia.ceph.com - ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6'] 2024-09-11T08:26:01.744 INFO:teuthology.run_tasks:Running task console_log... 2024-09-11T08:26:01.862 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f023e522e60>, signals=[15]) 2024-09-11T08:26:01.862 INFO:teuthology.run_tasks:Running task internal.connect... 2024-09-11T08:26:01.870 INFO:teuthology.task.internal:Opening connections... 2024-09-11T08:26:01.870 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T08:26:01.871 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:26:01.951 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T08:26:01.951 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:26:02.041 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-09-11T08:26:02.048 DEBUG:teuthology.orchestra.run.smithi073:> uname -m 2024-09-11T08:26:02.053 INFO:teuthology.orchestra.run.smithi073.stdout:x86_64 2024-09-11T08:26:02.053 DEBUG:teuthology.orchestra.run.smithi073:> cat /etc/os-release 2024-09-11T08:26:02.097 INFO:teuthology.orchestra.run.smithi073.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T08:26:02.097 INFO:teuthology.orchestra.run.smithi073.stdout:NAME="Ubuntu" 2024-09-11T08:26:02.097 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_ID="22.04" 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:VERSION_CODENAME=jammy 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:ID=ubuntu 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:ID_LIKE=debian 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T08:26:02.098 INFO:teuthology.orchestra.run.smithi073.stdout:UBUNTU_CODENAME=jammy 2024-09-11T08:26:02.099 INFO:teuthology.lock.ops:Updating smithi073.front.sepia.ceph.com on lock server 2024-09-11T08:26:02.118 DEBUG:teuthology.orchestra.run.smithi077:> uname -m 2024-09-11T08:26:02.123 INFO:teuthology.orchestra.run.smithi077.stdout:x86_64 2024-09-11T08:26:02.123 DEBUG:teuthology.orchestra.run.smithi077:> cat /etc/os-release 2024-09-11T08:26:02.168 INFO:teuthology.orchestra.run.smithi077.stdout:PRETTY_NAME="Ubuntu 22.04.1 LTS" 2024-09-11T08:26:02.168 INFO:teuthology.orchestra.run.smithi077.stdout:NAME="Ubuntu" 2024-09-11T08:26:02.168 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_ID="22.04" 2024-09-11T08:26:02.168 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION="22.04.1 LTS (Jammy Jellyfish)" 2024-09-11T08:26:02.168 INFO:teuthology.orchestra.run.smithi077.stdout:VERSION_CODENAME=jammy 2024-09-11T08:26:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:ID=ubuntu 2024-09-11T08:26:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:ID_LIKE=debian 2024-09-11T08:26:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:HOME_URL="https://www.ubuntu.com/" 2024-09-11T08:26:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2024-09-11T08:26:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2024-09-11T08:26:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2024-09-11T08:26:02.169 INFO:teuthology.orchestra.run.smithi077.stdout:UBUNTU_CODENAME=jammy 2024-09-11T08:26:02.169 INFO:teuthology.lock.ops:Updating smithi077.front.sepia.ceph.com on lock server 2024-09-11T08:26:02.193 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-09-11T08:26:02.203 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-09-11T08:26:02.210 INFO:teuthology.task.internal:Checking for old test directory... 2024-09-11T08:26:02.211 DEBUG:teuthology.orchestra.run.smithi073:> test '!' -e /home/ubuntu/cephtest 2024-09-11T08:26:02.213 DEBUG:teuthology.orchestra.run.smithi077:> test '!' -e /home/ubuntu/cephtest 2024-09-11T08:26:02.217 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-09-11T08:26:02.226 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-09-11T08:26:02.226 DEBUG:teuthology.orchestra.run.smithi073:> test -z $(ls -A /var/lib/ceph) 2024-09-11T08:26:02.259 DEBUG:teuthology.orchestra.run.smithi077:> test -z $(ls -A /var/lib/ceph) 2024-09-11T08:26:02.326 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-09-11T08:26:02.382 INFO:teuthology.run_tasks:Running task kernel... 2024-09-11T08:26:02.401 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-09-11T08:26:02.402 INFO:teuthology.task.kernel:config {'cluster1.mon.a': {'kdb': 1, 'sha1': 'distro'}, 'cluster2.mon.a': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-09-11T08:26:02.402 DEBUG:teuthology.orchestra.run.smithi073:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T08:26:02.402 DEBUG:teuthology.orchestra.run.smithi077:> test -f /run/.containerenv -o -f /.dockerenv 2024-09-11T08:26:02.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.407 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-09-11T08:26:02.408 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:26:02.408 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-09-11T08:26:02.454 INFO:teuthology.orchestra.run.smithi073.stdout:5.15.0-56-generic 2024-09-11T08:26:02.454 INFO:teuthology.task.kernel:Running kernel on smithi073: 5.15.0-56-generic 2024-09-11T08:26:02.455 DEBUG:teuthology.orchestra.run.smithi073:> sudo apt-get clean 2024-09-11T08:26:02.457 INFO:teuthology.orchestra.run.smithi077.stdout:5.15.0-56-generic 2024-09-11T08:26:02.457 INFO:teuthology.task.kernel:Running kernel on smithi077: 5.15.0-56-generic 2024-09-11T08:26:02.458 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get clean 2024-09-11T08:26:02.585 DEBUG:teuthology.orchestra.run.smithi073:> sudo apt-get update 2024-09-11T08:26:02.602 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2024-09-11T08:26:02.845 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T08:26:02.845 INFO:teuthology.orchestra.run.smithi073.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T08:26:02.895 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T08:26:02.903 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://security.ubuntu.com/ubuntu jammy-security InRelease [129 kB] 2024-09-11T08:26:02.907 INFO:teuthology.orchestra.run.smithi073.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T08:26:02.951 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease [128 kB] 2024-09-11T08:26:03.263 INFO:teuthology.orchestra.run.smithi073.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T08:26:03.282 INFO:teuthology.orchestra.run.smithi073.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T08:26:03.307 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease [127 kB] 2024-09-11T08:26:03.323 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T08:26:03.401 INFO:teuthology.orchestra.run.smithi073.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T08:26:03.446 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main i386 Packages [690 kB] 2024-09-11T08:26:03.604 INFO:teuthology.orchestra.run.smithi073.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T08:26:03.614 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://security.ubuntu.com/ubuntu jammy-security/main amd64 Packages [1,771 kB] 2024-09-11T08:26:03.641 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 Packages [1,988 kB] 2024-09-11T08:26:03.699 INFO:teuthology.orchestra.run.smithi073.stdout:Get:8 http://security.ubuntu.com/ubuntu jammy-security/main i386 Packages [533 kB] 2024-09-11T08:26:03.715 INFO:teuthology.orchestra.run.smithi073.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T08:26:03.724 INFO:teuthology.orchestra.run.smithi073.stdout:Get:10 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T08:26:03.725 INFO:teuthology.orchestra.run.smithi073.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T08:26:03.753 INFO:teuthology.orchestra.run.smithi073.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T08:26:03.767 INFO:teuthology.orchestra.run.smithi073.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T08:26:03.768 INFO:teuthology.orchestra.run.smithi073.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T08:26:03.786 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://security.ubuntu.com/ubuntu jammy-security/main Translation-en [291 kB] 2024-09-11T08:26:03.798 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy-updates/main Translation-en [349 kB] 2024-09-11T08:26:03.799 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 http://security.ubuntu.com/ubuntu jammy-security/main amd64 c-n-f Metadata [13.3 kB] 2024-09-11T08:26:03.800 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T08:26:03.801 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 Packages [2,327 kB] 2024-09-11T08:26:03.813 INFO:teuthology.orchestra.run.smithi077.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 c-n-f Metadata [17.8 kB] 2024-09-11T08:26:03.813 INFO:teuthology.orchestra.run.smithi077.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T08:26:03.815 INFO:teuthology.orchestra.run.smithi077.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 Packages [2,386 kB] 2024-09-11T08:26:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Get:15 http://security.ubuntu.com/ubuntu jammy-security/restricted i386 Packages [37.7 kB] 2024-09-11T08:26:03.821 INFO:teuthology.orchestra.run.smithi073.stdout:Get:16 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T08:26:03.833 INFO:teuthology.orchestra.run.smithi073.stdout:Get:17 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T08:26:03.847 INFO:teuthology.orchestra.run.smithi073.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy-updates/restricted i386 Packages [39.3 kB] 2024-09-11T08:26:03.847 INFO:teuthology.orchestra.run.smithi073.stdout:Get:19 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T08:26:03.848 INFO:teuthology.orchestra.run.smithi073.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T08:26:03.856 INFO:teuthology.orchestra.run.smithi073.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T08:26:03.857 INFO:teuthology.orchestra.run.smithi073.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T08:26:03.858 INFO:teuthology.orchestra.run.smithi073.stdout:Get:23 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T08:26:03.874 INFO:teuthology.orchestra.run.smithi073.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T08:26:03.881 INFO:teuthology.orchestra.run.smithi073.stdout:Get:25 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T08:26:03.881 INFO:teuthology.orchestra.run.smithi073.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T08:26:03.882 INFO:teuthology.orchestra.run.smithi073.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T08:26:03.886 INFO:teuthology.orchestra.run.smithi073.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T08:26:03.905 INFO:teuthology.orchestra.run.smithi073.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T08:26:03.911 INFO:teuthology.orchestra.run.smithi073.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T08:26:03.918 INFO:teuthology.orchestra.run.smithi077.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/restricted Translation-en [410 kB] 2024-09-11T08:26:03.918 INFO:teuthology.orchestra.run.smithi077.stdout:Get:18 http://security.ubuntu.com/ubuntu jammy-security/restricted Translation-en [400 kB] 2024-09-11T08:26:03.918 INFO:teuthology.orchestra.run.smithi073.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T08:26:03.918 INFO:teuthology.orchestra.run.smithi073.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T08:26:03.922 INFO:teuthology.orchestra.run.smithi077.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy-updates/restricted amd64 c-n-f Metadata [616 B] 2024-09-11T08:26:03.922 INFO:teuthology.orchestra.run.smithi077.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 Packages [1,123 kB] 2024-09-11T08:26:03.926 INFO:teuthology.orchestra.run.smithi077.stdout:Get:21 http://security.ubuntu.com/ubuntu jammy-security/restricted amd64 c-n-f Metadata [584 B] 2024-09-11T08:26:03.926 INFO:teuthology.orchestra.run.smithi077.stdout:Get:22 http://security.ubuntu.com/ubuntu jammy-security/universe i386 Packages [626 kB] 2024-09-11T08:26:03.933 INFO:teuthology.orchestra.run.smithi077.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy-updates/universe i386 Packages [731 kB] 2024-09-11T08:26:03.939 INFO:teuthology.orchestra.run.smithi077.stdout:Get:24 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 Packages [901 kB] 2024-09-11T08:26:03.952 INFO:teuthology.orchestra.run.smithi077.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy-updates/universe Translation-en [261 kB] 2024-09-11T08:26:03.954 INFO:teuthology.orchestra.run.smithi077.stdout:Get:26 http://security.ubuntu.com/ubuntu jammy-security/universe Translation-en [176 kB] 2024-09-11T08:26:03.957 INFO:teuthology.orchestra.run.smithi077.stdout:Get:27 http://security.ubuntu.com/ubuntu jammy-security/universe amd64 c-n-f Metadata [19.2 kB] 2024-09-11T08:26:03.958 INFO:teuthology.orchestra.run.smithi077.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 c-n-f Metadata [26.2 kB] 2024-09-11T08:26:03.958 INFO:teuthology.orchestra.run.smithi077.stdout:Get:29 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 Packages [37.2 kB] 2024-09-11T08:26:03.958 INFO:teuthology.orchestra.run.smithi077.stdout:Get:30 http://security.ubuntu.com/ubuntu jammy-security/multiverse i386 Packages [1,356 B] 2024-09-11T08:26:03.959 INFO:teuthology.orchestra.run.smithi077.stdout:Get:31 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T08:26:03.960 INFO:teuthology.orchestra.run.smithi073.stdout:Get:33 http://security.ubuntu.com/ubuntu jammy-security/multiverse Translation-en [7,588 B] 2024-09-11T08:26:03.961 INFO:teuthology.orchestra.run.smithi073.stdout:Get:34 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T08:26:03.963 INFO:teuthology.orchestra.run.smithi077.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 Packages [43.3 kB] 2024-09-11T08:26:03.964 INFO:teuthology.orchestra.run.smithi077.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse i386 Packages [4,752 B] 2024-09-11T08:26:03.964 INFO:teuthology.orchestra.run.smithi077.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T08:26:03.990 INFO:teuthology.orchestra.run.smithi073.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse Translation-en [10.8 kB] 2024-09-11T08:26:03.990 INFO:teuthology.orchestra.run.smithi073.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T08:26:03.990 INFO:teuthology.orchestra.run.smithi073.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T08:26:03.992 INFO:teuthology.orchestra.run.smithi073.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T08:26:03.993 INFO:teuthology.orchestra.run.smithi073.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T08:26:03.999 INFO:teuthology.orchestra.run.smithi073.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T08:26:04.000 INFO:teuthology.orchestra.run.smithi073.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T08:26:04.000 INFO:teuthology.orchestra.run.smithi073.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T08:26:04.032 INFO:teuthology.orchestra.run.smithi077.stdout:Get:35 http://security.ubuntu.com/ubuntu jammy-security/multiverse amd64 c-n-f Metadata [228 B] 2024-09-11T08:26:04.040 INFO:teuthology.orchestra.run.smithi077.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy-updates/multiverse amd64 c-n-f Metadata [444 B] 2024-09-11T08:26:04.040 INFO:teuthology.orchestra.run.smithi077.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 Packages [67.8 kB] 2024-09-11T08:26:04.041 INFO:teuthology.orchestra.run.smithi077.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy-backports/main i386 Packages [59.9 kB] 2024-09-11T08:26:04.042 INFO:teuthology.orchestra.run.smithi077.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy-backports/main Translation-en [11.1 kB] 2024-09-11T08:26:04.042 INFO:teuthology.orchestra.run.smithi077.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy-backports/main amd64 c-n-f Metadata [388 B] 2024-09-11T08:26:04.044 INFO:teuthology.orchestra.run.smithi077.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 Packages [28.8 kB] 2024-09-11T08:26:04.045 INFO:teuthology.orchestra.run.smithi077.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy-backports/universe i386 Packages [17.3 kB] 2024-09-11T08:26:04.081 INFO:teuthology.orchestra.run.smithi073.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T08:26:04.081 INFO:teuthology.orchestra.run.smithi073.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T08:26:04.126 INFO:teuthology.orchestra.run.smithi077.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy-backports/universe Translation-en [16.5 kB] 2024-09-11T08:26:04.127 INFO:teuthology.orchestra.run.smithi077.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-backports/universe amd64 c-n-f Metadata [672 B] 2024-09-11T08:26:09.921 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 15.8 MB in 3s (5,842 kB/s) 2024-09-11T08:26:10.142 INFO:teuthology.orchestra.run.smithi073.stdout:Fetched 15.8 MB in 3s (5,862 kB/s) 2024-09-11T08:26:11.388 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-09-11T08:26:11.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T08:26:11.469 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-09-11T08:26:11.532 INFO:teuthology.orchestra.run.smithi073.stdout:Reading package lists... 2024-09-11T08:26:11.550 DEBUG:teuthology.orchestra.run.smithi073:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T08:26:11.603 INFO:teuthology.orchestra.run.smithi073.stdout:Reading package lists... 2024-09-11T08:26:11.654 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2024-09-11T08:26:11.655 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2024-09-11T08:26:11.789 INFO:teuthology.orchestra.run.smithi073.stdout:Building dependency tree... 2024-09-11T08:26:11.790 INFO:teuthology.orchestra.run.smithi073.stdout:Reading state information... 2024-09-11T08:26:11.818 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T08:26:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T08:26:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T08:26:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T08:26:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T08:26:11.819 INFO:teuthology.orchestra.run.smithi077.stdout: python2.7-minimal 2024-09-11T08:26:11.819 INFO:teuthology.orchestra.run.smithi077.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T08:26:11.820 INFO:teuthology.orchestra.run.smithi077.stdout:The following additional packages will be installed: 2024-09-11T08:26:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T08:26:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T08:26:11.821 INFO:teuthology.orchestra.run.smithi077.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T08:26:11.822 INFO:teuthology.orchestra.run.smithi077.stdout:Suggested packages: 2024-09-11T08:26:11.822 INFO:teuthology.orchestra.run.smithi077.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T08:26:11.822 INFO:teuthology.orchestra.run.smithi077.stdout:Recommended packages: 2024-09-11T08:26:11.822 INFO:teuthology.orchestra.run.smithi077.stdout: thermald 2024-09-11T08:26:11.869 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2024-09-11T08:26:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T08:26:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T08:26:11.869 INFO:teuthology.orchestra.run.smithi077.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T08:26:11.870 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages will be upgraded: 2024-09-11T08:26:11.870 INFO:teuthology.orchestra.run.smithi077.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T08:26:11.921 INFO:teuthology.orchestra.run.smithi077.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T08:26:11.921 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 113 MB of archives. 2024-09-11T08:26:11.921 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T08:26:11.921 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T08:26:11.954 INFO:teuthology.orchestra.run.smithi073.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T08:26:11.954 INFO:teuthology.orchestra.run.smithi073.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T08:26:11.954 INFO:teuthology.orchestra.run.smithi073.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T08:26:11.955 INFO:teuthology.orchestra.run.smithi073.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T08:26:11.955 INFO:teuthology.orchestra.run.smithi073.stdout: python2 python2-dev python2-minimal python2.7 python2.7-dev 2024-09-11T08:26:11.955 INFO:teuthology.orchestra.run.smithi073.stdout: python2.7-minimal 2024-09-11T08:26:11.955 INFO:teuthology.orchestra.run.smithi073.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T08:26:11.956 INFO:teuthology.orchestra.run.smithi073.stdout:The following additional packages will be installed: 2024-09-11T08:26:11.956 INFO:teuthology.orchestra.run.smithi073.stdout: linux-generic linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T08:26:11.956 INFO:teuthology.orchestra.run.smithi073.stdout: linux-headers-generic linux-image-5.15.0-119-generic 2024-09-11T08:26:11.956 INFO:teuthology.orchestra.run.smithi073.stdout: linux-modules-5.15.0-119-generic linux-modules-extra-5.15.0-119-generic 2024-09-11T08:26:11.958 INFO:teuthology.orchestra.run.smithi073.stdout:Suggested packages: 2024-09-11T08:26:11.958 INFO:teuthology.orchestra.run.smithi073.stdout: fdutils linux-doc | linux-source-5.15.0 linux-tools 2024-09-11T08:26:11.958 INFO:teuthology.orchestra.run.smithi073.stdout:Recommended packages: 2024-09-11T08:26:11.958 INFO:teuthology.orchestra.run.smithi073.stdout: thermald 2024-09-11T08:26:12.014 INFO:teuthology.orchestra.run.smithi073.stdout:The following NEW packages will be installed: 2024-09-11T08:26:12.014 INFO:teuthology.orchestra.run.smithi073.stdout: linux-headers-5.15.0-119 linux-headers-5.15.0-119-generic 2024-09-11T08:26:12.014 INFO:teuthology.orchestra.run.smithi073.stdout: linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic 2024-09-11T08:26:12.014 INFO:teuthology.orchestra.run.smithi073.stdout: linux-modules-extra-5.15.0-119-generic 2024-09-11T08:26:12.015 INFO:teuthology.orchestra.run.smithi073.stdout:The following packages will be upgraded: 2024-09-11T08:26:12.016 INFO:teuthology.orchestra.run.smithi073.stdout: linux-generic linux-headers-generic linux-image-generic 2024-09-11T08:26:12.067 INFO:teuthology.orchestra.run.smithi073.stdout:3 upgraded, 5 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T08:26:12.067 INFO:teuthology.orchestra.run.smithi073.stdout:Need to get 113 MB of archives. 2024-09-11T08:26:12.067 INFO:teuthology.orchestra.run.smithi073.stdout:After this operation, 583 MB of additional disk space will be used. 2024-09-11T08:26:12.067 INFO:teuthology.orchestra.run.smithi073.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-5.15.0-119-generic amd64 5.15.0-119.129 [22.6 MB] 2024-09-11T08:26:12.359 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T08:26:12.456 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T08:26:12.507 INFO:teuthology.orchestra.run.smithi073.stdout:Get:2 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-5.15.0-119-generic amd64 5.15.0-119.129 [11.5 MB] 2024-09-11T08:26:12.618 INFO:teuthology.orchestra.run.smithi073.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-modules-extra-5.15.0-119-generic amd64 5.15.0-119.129 [63.9 MB] 2024-09-11T08:26:13.251 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T08:26:13.251 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T08:26:13.251 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T08:26:13.372 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T08:26:13.437 INFO:teuthology.orchestra.run.smithi073.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-generic amd64 5.15.0.119.119 [1,694 B] 2024-09-11T08:26:13.437 INFO:teuthology.orchestra.run.smithi073.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-image-generic amd64 5.15.0.119.119 [2,502 B] 2024-09-11T08:26:13.437 INFO:teuthology.orchestra.run.smithi073.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119 all 5.15.0-119.129 [12.3 MB] 2024-09-11T08:26:13.473 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T08:26:13.558 INFO:teuthology.orchestra.run.smithi073.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-5.15.0-119-generic amd64 5.15.0-119.129 [2,879 kB] 2024-09-11T08:26:13.580 INFO:teuthology.orchestra.run.smithi073.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 linux-headers-generic amd64 5.15.0.119.119 [2,350 B] 2024-09-11T08:26:13.859 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 113 MB in 2s (71.0 MB/s) 2024-09-11T08:26:13.987 INFO:teuthology.orchestra.run.smithi073.stdout:Fetched 113 MB in 2s (72.8 MB/s) 2024-09-11T08:26:14.059 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T08:26:14.162 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package linux-modules-5.15.0-119-generic. 2024-09-11T08:26:16.120 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T08:26:16.124 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:16.241 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:16.292 INFO:teuthology.orchestra.run.smithi073.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 129542 files and directories currently installed.) 2024-09-11T08:26:16.297 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../0-linux-modules-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:16.427 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:18.393 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T08:26:18.414 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:18.494 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:18.671 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package linux-image-5.15.0-119-generic. 2024-09-11T08:26:18.692 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../1-linux-image-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:18.785 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:18.955 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T08:26:18.975 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:19.013 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:19.207 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package linux-modules-extra-5.15.0-119-generic. 2024-09-11T08:26:19.219 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../2-linux-modules-extra-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:19.265 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:25.829 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T08:26:25.954 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T08:26:26.254 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T08:26:26.266 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../3-linux-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T08:26:26.382 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T08:26:26.418 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T08:26:26.711 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T08:26:26.733 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T08:26:26.759 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../4-linux-image-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T08:26:26.777 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T08:26:26.864 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-image-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T08:26:27.142 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-headers-5.15.0-119. 2024-09-11T08:26:27.165 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../5-linux-headers-5.15.0-119_5.15.0-119.129_all.deb ... 2024-09-11T08:26:27.208 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T08:26:32.246 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T08:26:32.269 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:32.313 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:32.729 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package linux-headers-5.15.0-119-generic. 2024-09-11T08:26:32.766 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../6-linux-headers-5.15.0-119-generic_5.15.0-119.129_amd64.deb ... 2024-09-11T08:26:32.804 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:35.008 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T08:26:35.647 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T08:26:36.000 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T08:26:36.113 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:36.116 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../7-linux-headers-generic_5.15.0.119.119_amd64.deb ... 2024-09-11T08:26:36.214 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T08:26:36.267 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking linux-headers-generic (5.15.0.119.119) over (5.15.0.56.54) ... 2024-09-11T08:26:36.335 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:36.618 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-headers-5.15.0-119 (5.15.0-119.129) ... 2024-09-11T08:26:36.739 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-headers-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:36.865 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-headers-generic (5.15.0.119.119) ... 2024-09-11T08:26:37.019 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:37.661 INFO:teuthology.orchestra.run.smithi073.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T08:26:37.661 INFO:teuthology.orchestra.run.smithi073.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T08:26:37.661 INFO:teuthology.orchestra.run.smithi073.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T08:26:37.661 INFO:teuthology.orchestra.run.smithi073.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T08:26:37.836 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:38.461 INFO:teuthology.orchestra.run.smithi077.stdout:I: /vmlinuz.old is now a symlink to boot/vmlinuz-5.15.0-56-generic 2024-09-11T08:26:38.461 INFO:teuthology.orchestra.run.smithi077.stdout:I: /initrd.img.old is now a symlink to boot/initrd.img-5.15.0-56-generic 2024-09-11T08:26:38.461 INFO:teuthology.orchestra.run.smithi077.stdout:I: /vmlinuz is now a symlink to boot/vmlinuz-5.15.0-119-generic 2024-09-11T08:26:38.461 INFO:teuthology.orchestra.run.smithi077.stdout:I: /initrd.img is now a symlink to boot/initrd.img-5.15.0-119-generic 2024-09-11T08:26:39.305 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-modules-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:39.529 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:40.740 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-modules-extra-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:40.924 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T08:26:41.040 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T08:26:41.158 INFO:teuthology.orchestra.run.smithi073.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:41.214 INFO:teuthology.orchestra.run.smithi073.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T08:26:41.214 INFO:teuthology.orchestra.run.smithi073.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T08:26:42.175 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-image-generic (5.15.0.119.119) ... 2024-09-11T08:26:42.301 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up linux-generic (5.15.0.119.119) ... 2024-09-11T08:26:42.427 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for linux-image-5.15.0-119-generic (5.15.0-119.129) ... 2024-09-11T08:26:42.495 INFO:teuthology.orchestra.run.smithi077.stdout:/etc/kernel/postinst.d/initramfs-tools: 2024-09-11T08:26:42.495 INFO:teuthology.orchestra.run.smithi077.stdout:update-initramfs: Generating /boot/initrd.img-5.15.0-119-generic 2024-09-11T08:26:55.588 INFO:teuthology.orchestra.run.smithi073.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T08:26:55.588 INFO:teuthology.orchestra.run.smithi073.stdout:Sourcing file `/etc/default/grub' 2024-09-11T08:26:55.601 INFO:teuthology.orchestra.run.smithi073.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T08:26:55.616 INFO:teuthology.orchestra.run.smithi073.stdout:Generating grub configuration file ... 2024-09-11T08:26:55.915 INFO:teuthology.orchestra.run.smithi073.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T08:26:55.937 INFO:teuthology.orchestra.run.smithi073.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T08:26:56.342 INFO:teuthology.orchestra.run.smithi073.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T08:26:56.349 INFO:teuthology.orchestra.run.smithi073.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T08:26:56.431 INFO:teuthology.orchestra.run.smithi073.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T08:26:56.436 INFO:teuthology.orchestra.run.smithi073.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T08:26:56.550 INFO:teuthology.orchestra.run.smithi077.stdout:/etc/kernel/postinst.d/zz-update-grub: 2024-09-11T08:26:56.550 INFO:teuthology.orchestra.run.smithi077.stdout:Sourcing file `/etc/default/grub' 2024-09-11T08:26:56.569 INFO:teuthology.orchestra.run.smithi077.stdout:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T08:26:56.593 INFO:teuthology.orchestra.run.smithi073.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T08:26:56.594 INFO:teuthology.orchestra.run.smithi073.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T08:26:56.594 INFO:teuthology.orchestra.run.smithi073.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T08:26:56.595 INFO:teuthology.orchestra.run.smithi077.stdout:Generating grub configuration file ... 2024-09-11T08:26:56.638 INFO:teuthology.orchestra.run.smithi073.stdout:done 2024-09-11T08:26:56.887 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T08:26:56.911 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T08:26:57.119 DEBUG:teuthology.orchestra.run.smithi073:> dpkg -s linux-image-generic 2024-09-11T08:26:57.139 INFO:teuthology.orchestra.run.smithi073.stdout:Package: linux-image-generic 2024-09-11T08:26:57.139 INFO:teuthology.orchestra.run.smithi073.stdout:Status: install ok installed 2024-09-11T08:26:57.139 INFO:teuthology.orchestra.run.smithi073.stdout:Priority: optional 2024-09-11T08:26:57.139 INFO:teuthology.orchestra.run.smithi073.stdout:Section: kernel 2024-09-11T08:26:57.139 INFO:teuthology.orchestra.run.smithi073.stdout:Installed-Size: 21 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: amd64 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Source: linux-meta 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Version: 5.15.0.119.119 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Recommends: thermald 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout:Description: Generic Linux kernel image 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout: This package will always depend on the latest generic kernel image 2024-09-11T08:26:57.140 INFO:teuthology.orchestra.run.smithi073.stdout: available. 2024-09-11T08:26:57.140 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T08:26:57.141 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T08:26:57.141 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T08:26:57.141 INFO:teuthology.task.kernel:Installing distro kernel on cluster1.mon.a... 2024-09-11T08:26:57.141 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi073.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T08:26:57.141 DEBUG:teuthology.orchestra.run.smithi073:> sudo apt-get clean 2024-09-11T08:26:57.231 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T08:26:57.236 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T08:26:57.298 INFO:teuthology.orchestra.run.smithi077.stdout:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T08:26:57.304 INFO:teuthology.orchestra.run.smithi077.stdout:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T08:26:57.310 DEBUG:teuthology.orchestra.run.smithi073:> sudo apt-get update 2024-09-11T08:26:57.423 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T08:26:57.423 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T08:26:57.431 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T08:26:57.448 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T08:26:57.473 INFO:teuthology.orchestra.run.smithi077.stdout:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T08:26:57.474 INFO:teuthology.orchestra.run.smithi077.stdout:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T08:26:57.474 INFO:teuthology.orchestra.run.smithi077.stdout:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T08:26:57.513 INFO:teuthology.orchestra.run.smithi077.stdout:done 2024-09-11T08:26:57.998 DEBUG:teuthology.orchestra.run.smithi077:> dpkg -s linux-image-generic 2024-09-11T08:26:58.022 INFO:teuthology.orchestra.run.smithi077.stdout:Package: linux-image-generic 2024-09-11T08:26:58.022 INFO:teuthology.orchestra.run.smithi077.stdout:Status: install ok installed 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Priority: optional 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Section: kernel 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Installed-Size: 21 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: amd64 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Source: linux-meta 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Version: 5.15.0.119.119 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Recommends: thermald 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout:Description: Generic Linux kernel image 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout: This package will always depend on the latest generic kernel image 2024-09-11T08:26:58.023 INFO:teuthology.orchestra.run.smithi077.stdout: available. 2024-09-11T08:26:58.024 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.15.0-56-generic Expected: 5.15.0-119-generic 2024-09-11T08:26:58.024 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-09-11T08:26:58.024 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-09-11T08:26:58.024 INFO:teuthology.task.kernel:Installing distro kernel on cluster2.mon.a... 2024-09-11T08:26:58.024 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi077.front.sepia.ceph.com, path=None, version=distro) 2024-09-11T08:26:58.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get clean 2024-09-11T08:26:58.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2024-09-11T08:26:58.516 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T08:26:58.516 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T08:26:58.587 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T08:26:58.668 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T08:26:58.718 INFO:teuthology.orchestra.run.smithi073.stdout:Reading package lists... 2024-09-11T08:26:58.737 DEBUG:teuthology.orchestra.run.smithi073:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T08:26:58.799 INFO:teuthology.orchestra.run.smithi073.stdout:Reading package lists... 2024-09-11T08:26:59.016 INFO:teuthology.orchestra.run.smithi073.stdout:Building dependency tree... 2024-09-11T08:26:59.016 INFO:teuthology.orchestra.run.smithi073.stdout:Reading state information... 2024-09-11T08:26:59.199 INFO:teuthology.orchestra.run.smithi073.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T08:26:59.199 INFO:teuthology.orchestra.run.smithi073.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T08:26:59.199 INFO:teuthology.orchestra.run.smithi073.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T08:26:59.199 INFO:teuthology.orchestra.run.smithi073.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T08:26:59.199 INFO:teuthology.orchestra.run.smithi073.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T08:26:59.200 INFO:teuthology.orchestra.run.smithi073.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T08:26:59.200 INFO:teuthology.orchestra.run.smithi073.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T08:26:59.200 INFO:teuthology.orchestra.run.smithi073.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T08:26:59.241 INFO:teuthology.orchestra.run.smithi073.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T08:26:59.244 DEBUG:teuthology.orchestra.run.smithi073:> dpkg -s linux-image-generic 2024-09-11T08:26:59.258 INFO:teuthology.orchestra.run.smithi073.stdout:Package: linux-image-generic 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Status: install ok installed 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Priority: optional 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Section: kernel 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Installed-Size: 21 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Architecture: amd64 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Source: linux-meta 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Version: 5.15.0.119.119 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Recommends: thermald 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout:Description: Generic Linux kernel image 2024-09-11T08:26:59.259 INFO:teuthology.orchestra.run.smithi073.stdout: This package will always depend on the latest generic kernel image 2024-09-11T08:26:59.260 INFO:teuthology.orchestra.run.smithi073.stdout: available. 2024-09-11T08:26:59.260 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-09-11T08:26:59.304 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.mdKc3RJYol 2024-09-11T08:26:59.305 DEBUG:teuthology.orchestra.run.smithi073:> sudo cp /boot/grub/grub.cfg /tmp/tmp.mdKc3RJYol 2024-09-11T08:26:59.359 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.mdKc3RJYol 2024-09-11T08:26:59.469 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.mdKc3RJYol is 10KB 2024-09-11T08:26:59.482 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.mdKc3RJYol 2024-09-11T08:26:59.487 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T08:26:59.540 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:26:59.540 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T08:26:59.540 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T08:26:59.604 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T08:26:59.604 DEBUG:teuthology.orchestra.run.smithi073:> sudo update-grub 2024-09-11T08:26:59.923 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-09-11T08:26:59.942 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2024-09-11T08:27:00.003 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-09-11T08:27:00.210 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2024-09-11T08:27:00.210 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2024-09-11T08:27:00.393 INFO:teuthology.orchestra.run.smithi077.stdout:linux-image-generic is already the newest version (5.15.0.119.119). 2024-09-11T08:27:00.393 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T08:27:00.393 INFO:teuthology.orchestra.run.smithi077.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T08:27:00.393 INFO:teuthology.orchestra.run.smithi077.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T08:27:00.393 INFO:teuthology.orchestra.run.smithi077.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T08:27:00.394 INFO:teuthology.orchestra.run.smithi077.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T08:27:00.394 INFO:teuthology.orchestra.run.smithi077.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T08:27:00.394 INFO:teuthology.orchestra.run.smithi077.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T08:27:00.434 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 0 newly installed, 0 to remove and 344 not upgraded. 2024-09-11T08:27:00.436 DEBUG:teuthology.orchestra.run.smithi077:> dpkg -s linux-image-generic 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Package: linux-image-generic 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Status: install ok installed 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Priority: optional 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Section: kernel 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Installed-Size: 21 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Maintainer: Ubuntu Kernel Team 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Architecture: amd64 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Source: linux-meta 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Version: 5.15.0.119.119 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Provides: spl-modules (= 2.1.5-1ubuntu6~22.04.4), v4l2loopback-modules (= 0.12.7-2ubuntu2~22.04.1), virtualbox-guest-modules (= 5.15.0-119), wireguard-modules (= 1.0.0), zfs-modules (= 2.1.5-1ubuntu6~22.04.4) 2024-09-11T08:27:00.451 INFO:teuthology.orchestra.run.smithi077.stdout:Depends: linux-image-5.15.0-119-generic, linux-modules-extra-5.15.0-119-generic, linux-firmware, intel-microcode, amd64-microcode 2024-09-11T08:27:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:Recommends: thermald 2024-09-11T08:27:00.452 INFO:teuthology.orchestra.run.smithi077.stdout:Description: Generic Linux kernel image 2024-09-11T08:27:00.452 INFO:teuthology.orchestra.run.smithi077.stdout: This package will always depend on the latest generic kernel image 2024-09-11T08:27:00.452 INFO:teuthology.orchestra.run.smithi077.stdout: available. 2024-09-11T08:27:00.452 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-09-11T08:27:00.500 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.leoRNB3gSY 2024-09-11T08:27:00.500 DEBUG:teuthology.orchestra.run.smithi077:> sudo cp /boot/grub/grub.cfg /tmp/tmp.leoRNB3gSY 2024-09-11T08:27:00.554 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.leoRNB3gSY 2024-09-11T08:27:00.673 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.leoRNB3gSY is 10KB 2024-09-11T08:27:00.685 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.leoRNB3gSY 2024-09-11T08:27:00.690 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/grub.d/01_ceph_kernel 2024-09-11T08:27:00.743 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:27:00.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/grub.d/01_ceph_kernel 2024-09-11T08:27:00.743 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 755 /etc/grub.d/01_ceph_kernel 2024-09-11T08:27:00.807 INFO:teuthology.task.kernel:Distro Kernel Version: 5.15.0-119-generic 2024-09-11T08:27:00.808 DEBUG:teuthology.orchestra.run.smithi077:> sudo update-grub 2024-09-11T08:27:00.836 INFO:teuthology.orchestra.run.smithi073.stderr:Sourcing file `/etc/default/grub' 2024-09-11T08:27:00.837 INFO:teuthology.orchestra.run.smithi073.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T08:27:00.839 INFO:teuthology.orchestra.run.smithi073.stderr:Generating grub configuration file ... 2024-09-11T08:27:01.090 INFO:teuthology.orchestra.run.smithi073.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T08:27:01.096 INFO:teuthology.orchestra.run.smithi073.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T08:27:01.390 INFO:teuthology.orchestra.run.smithi073.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T08:27:01.395 INFO:teuthology.orchestra.run.smithi073.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T08:27:01.437 INFO:teuthology.orchestra.run.smithi073.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T08:27:01.442 INFO:teuthology.orchestra.run.smithi073.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T08:27:01.571 INFO:teuthology.orchestra.run.smithi073.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T08:27:01.571 INFO:teuthology.orchestra.run.smithi073.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T08:27:01.571 INFO:teuthology.orchestra.run.smithi073.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T08:27:01.582 INFO:teuthology.orchestra.run.smithi073.stderr:done 2024-09-11T08:27:01.595 DEBUG:teuthology.orchestra.run.smithi073:> sudo shutdown -r now 2024-09-11T08:27:02.203 INFO:teuthology.orchestra.run.smithi077.stderr:Sourcing file `/etc/default/grub' 2024-09-11T08:27:02.205 INFO:teuthology.orchestra.run.smithi077.stderr:Sourcing file `/etc/default/grub.d/init-select.cfg' 2024-09-11T08:27:02.208 INFO:teuthology.orchestra.run.smithi077.stderr:Generating grub configuration file ... 2024-09-11T08:27:02.414 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.15.0-119-generic 2024-09-11T08:27:02.420 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.15.0-119-generic 2024-09-11T08:27:02.708 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.15.0-56-generic 2024-09-11T08:27:02.713 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.15.0-56-generic 2024-09-11T08:27:02.755 INFO:teuthology.orchestra.run.smithi077.stderr:Found linux image: /boot/vmlinuz-5.4.0-135-generic 2024-09-11T08:27:02.760 INFO:teuthology.orchestra.run.smithi077.stderr:Found initrd image: /boot/initrd.img-5.4.0-135-generic 2024-09-11T08:27:02.905 INFO:teuthology.orchestra.run.smithi077.stderr:Warning: os-prober will not be executed to detect other bootable partitions. 2024-09-11T08:27:02.905 INFO:teuthology.orchestra.run.smithi077.stderr:Systems on them will not be added to the GRUB boot configuration. 2024-09-11T08:27:02.905 INFO:teuthology.orchestra.run.smithi077.stderr:Check GRUB_DISABLE_OS_PROBER documentation entry. 2024-09-11T08:27:02.927 INFO:teuthology.orchestra.run.smithi077.stderr:done 2024-09-11T08:27:02.929 DEBUG:teuthology.orchestra.run.smithi077:> sudo shutdown -r now 2024-09-11T08:27:31.625 INFO:teuthology.task.kernel:Checking client cluster1.mon.a for new kernel version... 2024-09-11T08:27:31.626 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-09-11T08:27:31.628 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:27:32.932 INFO:teuthology.task.kernel:Checking client cluster2.mon.a for new kernel version... 2024-09-11T08:27:32.933 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-09-11T08:27:32.933 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:27:49.949 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-09-11T08:27:51.485 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-09-11T08:27:58.957 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-09-11T08:27:58.958 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:28:00.487 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-09-11T08:28:00.488 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:28:02.013 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-09-11T08:28:03.553 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-09-11T08:28:14.025 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-09-11T08:28:14.026 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:28:15.555 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-09-11T08:28:15.556 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:28:32.445 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.73 2024-09-11T08:28:33.981 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.77 2024-09-11T08:28:47.458 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-09-11T08:28:47.459 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:28:47.776 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-09-11T08:28:48.381 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-09-11T08:28:48.381 INFO:teuthology.task.kernel:Checking kernel version of cluster1.mon.a, want "5.15.0-119-generic"... 2024-09-11T08:28:48.381 DEBUG:teuthology.orchestra.run.smithi073:> uname -r 2024-09-11T08:28:48.427 INFO:teuthology.orchestra.run.smithi073.stdout:5.15.0-119-generic 2024-09-11T08:28:48.428 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T08:28:48.428 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T08:28:48.428 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T08:28:48.981 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-09-11T08:28:48.982 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:28:49.201 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-09-11T08:28:49.429 INFO:teuthology.task.kernel:Enabling kdb on cluster1.mon.a... 2024-09-11T08:28:49.429 DEBUG:teuthology.orchestra.run.smithi073:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T08:28:49.757 INFO:teuthology.orchestra.run.smithi073.stdout:ttyS1 2024-09-11T08:28:49.785 DEBUG:teuthology.parallel:result is None 2024-09-11T08:28:49.820 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-09-11T08:28:49.821 INFO:teuthology.task.kernel:Checking kernel version of cluster2.mon.a, want "5.15.0-119-generic"... 2024-09-11T08:28:49.821 DEBUG:teuthology.orchestra.run.smithi077:> uname -r 2024-09-11T08:28:49.866 INFO:teuthology.orchestra.run.smithi077.stdout:5.15.0-119-generic 2024-09-11T08:28:49.867 DEBUG:teuthology.task.kernel:current kernel version is 5.15.0-119-generic vs 5.15.0-119-generic 2024-09-11T08:28:49.867 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-09-11T08:28:49.867 DEBUG:teuthology.task.kernel:Distro of this test job: ubuntu 2024-09-11T08:28:50.868 INFO:teuthology.task.kernel:Enabling kdb on cluster2.mon.a... 2024-09-11T08:28:50.868 DEBUG:teuthology.orchestra.run.smithi077:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-09-11T08:28:50.946 INFO:teuthology.orchestra.run.smithi077.stdout:ttyS1 2024-09-11T08:28:50.972 DEBUG:teuthology.parallel:result is None 2024-09-11T08:28:50.973 INFO:teuthology.run_tasks:Running task internal.base... 2024-09-11T08:28:50.980 INFO:teuthology.task.internal:Creating test directory... 2024-09-11T08:28:50.980 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T08:28:50.982 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-09-11T08:28:50.987 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-09-11T08:28:50.994 INFO:teuthology.run_tasks:Running task internal.archive... 2024-09-11T08:28:51.004 INFO:teuthology.task.internal:Creating archive directory... 2024-09-11T08:28:51.004 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T08:28:51.028 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-09-11T08:28:51.119 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-09-11T08:28:51.125 INFO:teuthology.task.internal:Enabling coredump saving... 2024-09-11T08:28:51.125 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T08:28:51.128 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-09-11T08:28:51.148 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:28:51.154 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:28:51.156 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:28:51.162 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-09-11T08:28:51.164 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-09-11T08:28:51.175 INFO:teuthology.task.internal:Configuring sudo... 2024-09-11T08:28:51.175 DEBUG:teuthology.orchestra.run.smithi073:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T08:28:51.201 DEBUG:teuthology.orchestra.run.smithi077:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-09-11T08:28:51.226 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-09-11T08:28:51.236 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-09-11T08:28:51.236 DEBUG:teuthology.orchestra.run.smithi073:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T08:28:51.256 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-09-11T08:28:51.277 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T08:28:51.307 DEBUG:teuthology.orchestra.run.smithi073:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T08:28:51.355 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:28:51.356 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T08:28:51.419 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-09-11T08:28:51.425 DEBUG:teuthology.orchestra.run.smithi077:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-09-11T08:28:51.471 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:28:51.471 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-09-11T08:28:51.537 DEBUG:teuthology.orchestra.run.smithi073:> sudo service rsyslog restart 2024-09-11T08:28:51.540 DEBUG:teuthology.orchestra.run.smithi077:> sudo service rsyslog restart 2024-09-11T08:28:51.612 INFO:teuthology.run_tasks:Running task internal.timer... 2024-09-11T08:28:51.619 INFO:teuthology.task.internal:Starting timer... 2024-09-11T08:28:51.619 INFO:teuthology.run_tasks:Running task pcp... 2024-09-11T08:28:51.630 INFO:teuthology.run_tasks:Running task selinux... 2024-09-11T08:28:51.638 DEBUG:teuthology.task.selinux:Excluding smithi073: OS 'ubuntu' does not support SELinux 2024-09-11T08:28:51.638 DEBUG:teuthology.task.selinux:Excluding smithi077: OS 'ubuntu' does not support SELinux 2024-09-11T08:28:51.638 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-09-11T08:28:51.638 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2024-09-11T08:28:51.639 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-09-11T08:28:51.639 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-09-11T08:28:51.647 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2024-09-11T08:28:51.650 INFO:teuthology.repo_utils:Fetching git.ceph.com_ceph-cm-ansible_main from origin 2024-09-11T08:28:51.772 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-09-11T08:28:51.783 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-09-11T08:28:51.784 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi073.front.sepia.ceph.com,smithi077.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-09-11T08:36:18.927 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi073.front.sepia.ceph.com'), Remote(name='ubuntu@smithi077.front.sepia.ceph.com')] 2024-09-11T08:36:18.929 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-09-11T08:36:18.930 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi073.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:36:19.015 DEBUG:teuthology.orchestra.run.smithi073:> true 2024-09-11T08:36:19.098 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi073.front.sepia.ceph.com' 2024-09-11T08:36:19.098 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-09-11T08:36:19.099 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi077.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-09-11T08:36:19.196 DEBUG:teuthology.orchestra.run.smithi077:> true 2024-09-11T08:36:19.273 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi077.front.sepia.ceph.com' 2024-09-11T08:36:19.273 INFO:teuthology.run_tasks:Running task clock... 2024-09-11T08:36:19.285 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-09-11T08:36:19.286 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T08:36:19.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:36:19.289 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-09-11T08:36:19.289 DEBUG:teuthology.orchestra.run.smithi077:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T08:36:19.315 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T08:36:19.315 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Command line: ntpd -gq 2024-09-11T08:36:19.315 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: ---------------------------------------------------- 2024-09-11T08:36:19.315 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T08:36:19.315 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T08:36:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: corporation. Support and training for ntp-4 are 2024-09-11T08:36:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: available at https://www.nwtime.org/support 2024-09-11T08:36:19.316 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: ---------------------------------------------------- 2024-09-11T08:36:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: proto: precision = 0.073 usec (-24) 2024-09-11T08:36:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: basedate set to 2022-02-04 2024-09-11T08:36:19.317 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: gps base set to 2022-02-06 (week 2196) 2024-09-11T08:36:19.318 INFO:teuthology.orchestra.run.smithi073.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T08:36:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T08:36:19.318 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T08:36:19.319 INFO:teuthology.orchestra.run.smithi073.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T08:36:19.319 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T08:36:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T08:36:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T08:36:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Listen normally on 3 enp3s0f1 172.21.15.73:123 2024-09-11T08:36:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Listen normally on 4 lo [::1]:123 2024-09-11T08:36:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:ce7d%5]:123 2024-09-11T08:36:19.320 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:19 ntpd[18463]: Listening on routing socket on fd #22 for interface updates 2024-09-11T08:36:19.340 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: ntpd 4.2.8p15@1.3728-o Wed Feb 16 17:13:02 UTC 2022 (1): Starting 2024-09-11T08:36:19.340 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Command line: ntpd -gq 2024-09-11T08:36:19.340 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: ---------------------------------------------------- 2024-09-11T08:36:19.340 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: ntp-4 is maintained by Network Time Foundation, 2024-09-11T08:36:19.340 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Inc. (NTF), a non-profit 501(c)(3) public-benefit 2024-09-11T08:36:19.341 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: corporation. Support and training for ntp-4 are 2024-09-11T08:36:19.341 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: available at https://www.nwtime.org/support 2024-09-11T08:36:19.341 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: ---------------------------------------------------- 2024-09-11T08:36:19.341 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: proto: precision = 0.062 usec (-24) 2024-09-11T08:36:19.341 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: basedate set to 2022-02-04 2024-09-11T08:36:19.341 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: gps base set to 2022-02-06 (week 2196) 2024-09-11T08:36:19.342 INFO:teuthology.orchestra.run.smithi077.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T08:36:19.342 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2024-09-11T08:36:19.342 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: restrict ::: KOD does nothing without LIMITED. 2024-09-11T08:36:19.342 INFO:teuthology.orchestra.run.smithi077.stderr:restrict ::: KOD does nothing without LIMITED. 2024-09-11T08:36:19.343 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Listen and drop on 0 v6wildcard [::]:123 2024-09-11T08:36:19.343 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2024-09-11T08:36:19.343 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Listen normally on 2 lo 127.0.0.1:123 2024-09-11T08:36:19.344 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Listen normally on 3 enp3s0f1 172.21.15.77:123 2024-09-11T08:36:19.344 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Listen normally on 4 lo [::1]:123 2024-09-11T08:36:19.344 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d3d3%5]:123 2024-09-11T08:36:19.344 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:19 ntpd[18431]: Listening on routing socket on fd #22 for interface updates 2024-09-11T08:36:20.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:20.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:20 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:20.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:20.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:20.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:20.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:20.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:20 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:21.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:21.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:21.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:21.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:21.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:21.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:21.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:21.319 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:21 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:22.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.319 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:22.319 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:22.319 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.319 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.320 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:22.320 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:22 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:22.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.343 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.343 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.343 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.343 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:22.343 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:22.344 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:22.344 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:22.344 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:22.344 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:22 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:23.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:23 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:23.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:23 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:23.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:23 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:23.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:23 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:23.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:23 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:23.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:23 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:23.340 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:23 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:23.340 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:23 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:23.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:23 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:23.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:23 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:23.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:23 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:23.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:23 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:24.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:24.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:24 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:24.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:24 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:25.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:25 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:25.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:25 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:25.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:25 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:25.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:25 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:25.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:25 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:25.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:25 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:25.340 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:25 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:25.340 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:25 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:25.340 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:25 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:25.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:25 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:25.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:25 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:25.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:25 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:26.316 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:26.317 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:26.317 INFO:teuthology.orchestra.run.smithi073.stdout:11 Sep 08:36:26 ntpd[18463]: ntpd: time slew -0.004158 s 2024-09-11T08:36:26.317 INFO:teuthology.orchestra.run.smithi073.stdout:ntpd: time slew -0.004158s 2024-09-11T08:36:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T08:36:26.318 INFO:teuthology.orchestra.run.smithi073.stderr:11 Sep 08:36:26 ntpd[18463]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T08:36:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2024-09-11T08:36:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: can't open /var/log/ntpstats/rawstats.20240911: Permission denied 2024-09-11T08:36:26.341 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2024-09-11T08:36:26.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: can't open /var/log/ntpstats/peerstats.20240911: Permission denied 2024-09-11T08:36:26.342 INFO:teuthology.orchestra.run.smithi077.stdout:11 Sep 08:36:26 ntpd[18431]: ntpd: time slew -0.002434 s 2024-09-11T08:36:26.342 INFO:teuthology.orchestra.run.smithi077.stdout:ntpd: time slew -0.002434s 2024-09-11T08:36:26.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: couldn't unlink /var/log/ntpstats/loopstats: Permission denied 2024-09-11T08:36:26.342 INFO:teuthology.orchestra.run.smithi077.stderr:11 Sep 08:36:26 ntpd[18431]: can't open /var/log/ntpstats/loopstats.20240911: Permission denied 2024-09-11T08:36:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T08:36:26.373 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================== 2024-09-11T08:36:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.373 INFO:teuthology.orchestra.run.smithi073.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.396 INFO:teuthology.orchestra.run.smithi077.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T08:36:26.396 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================== 2024-09-11T08:36:26.397 INFO:teuthology.orchestra.run.smithi077.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.397 INFO:teuthology.orchestra.run.smithi077.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.397 INFO:teuthology.orchestra.run.smithi077.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.397 INFO:teuthology.orchestra.run.smithi077.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 +0.000 0.000 2024-09-11T08:36:26.399 INFO:teuthology.run_tasks:Running task install... 2024-09-11T08:36:26.409 DEBUG:teuthology.task.install:project ceph 2024-09-11T08:36:26.409 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}} 2024-09-11T08:36:26.409 DEBUG:teuthology.task.install:config {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'} 2024-09-11T08:36:26.409 INFO:teuthology.task.install:Using flavor: default 2024-09-11T08:36:26.415 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'ceph-volume', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'ceph-volume', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-09-11T08:36:26.415 INFO:teuthology.task.install:extra packages: ['rbd-mirror'] 2024-09-11T08:36:26.415 DEBUG:teuthology.orchestra.run.smithi073:> sudo apt-key list | grep Ceph 2024-09-11T08:36:26.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-key list | grep Ceph 2024-09-11T08:36:26.478 INFO:teuthology.orchestra.run.smithi073.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T08:36:26.499 INFO:teuthology.orchestra.run.smithi077.stderr:Warning: apt-key is deprecated. Manage keyring files in trusted.gpg.d instead (see apt-key(8)). 2024-09-11T08:36:26.507 INFO:teuthology.orchestra.run.smithi073.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T08:36:26.507 INFO:teuthology.orchestra.run.smithi073.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T08:36:26.508 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-11T08:36:26.508 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T08:36:26.528 INFO:teuthology.orchestra.run.smithi077.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2024-09-11T08:36:26.528 INFO:teuthology.orchestra.run.smithi077.stdout:uid [ unknown] Ceph.com (release key) 2024-09-11T08:36:26.528 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, ceph-volume, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, rbd-mirror on remote deb x86_64 2024-09-11T08:36:26.528 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T08:36:26.648 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T08:36:26.648 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:36:26.663 INFO:teuthology.task.install.deb:Pulling from https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default/ 2024-09-11T08:36:26.663 INFO:teuthology.task.install.deb:Package version is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:36:26.761 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:36:26.761 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T08:36:26.772 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:36:26.772 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2024-09-11T08:36:26.776 DEBUG:teuthology.orchestra.run.smithi073:> sudo apt-get update 2024-09-11T08:36:26.786 DEBUG:teuthology.orchestra.run.smithi077:> sudo apt-get update 2024-09-11T08:36:26.965 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:1 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T08:36:26.965 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:2 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T08:36:26.973 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:3 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T08:36:26.990 INFO:teuthology.orchestra.run.smithi073.stdout:Hit:4 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T08:36:26.998 INFO:teuthology.orchestra.run.smithi073.stdout:Ign:5 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T08:36:27.008 INFO:teuthology.orchestra.run.smithi077.stdout:Ign:1 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy InRelease 2024-09-11T08:36:27.019 INFO:teuthology.orchestra.run.smithi073.stdout:Get:6 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T08:36:27.030 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release [7,662 B] 2024-09-11T08:36:27.044 INFO:teuthology.orchestra.run.smithi073.stdout:Ign:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T08:36:27.052 INFO:teuthology.orchestra.run.smithi077.stdout:Ign:3 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy Release.gpg 2024-09-11T08:36:27.062 INFO:teuthology.orchestra.run.smithi073.stdout:Get:8 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T08:36:27.073 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T08:36:27.089 INFO:teuthology.orchestra.run.smithi073.stdout:Get:9 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 Packages [17.9 kB] 2024-09-11T08:36:27.093 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:5 http://archive.ubuntu.com/ubuntu jammy InRelease 2024-09-11T08:36:27.093 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:6 http://security.ubuntu.com/ubuntu jammy-security InRelease 2024-09-11T08:36:27.106 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main i386 Packages [3,404 B] 2024-09-11T08:36:27.167 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:8 http://archive.ubuntu.com/ubuntu jammy-updates InRelease 2024-09-11T08:36:27.249 INFO:teuthology.orchestra.run.smithi077.stdout:Hit:9 http://archive.ubuntu.com/ubuntu jammy-backports InRelease 2024-09-11T08:36:27.325 INFO:teuthology.orchestra.run.smithi073.stdout:Fetched 29.0 kB in 0s (79.0 kB/s) 2024-09-11T08:36:27.464 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 29.0 kB in 0s (58.6 kB/s) 2024-09-11T08:36:28.341 INFO:teuthology.orchestra.run.smithi073.stdout:Reading package lists... 2024-09-11T08:36:28.357 DEBUG:teuthology.orchestra.run.smithi073:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy rbd-mirror=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:36:28.421 INFO:teuthology.orchestra.run.smithi073.stdout:Reading package lists... 2024-09-11T08:36:28.480 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-09-11T08:36:28.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=19.1.1-298-gb53b6d3e-1jammy cephadm=19.1.1-298-gb53b6d3e-1jammy ceph-mds=19.1.1-298-gb53b6d3e-1jammy ceph-mgr=19.1.1-298-gb53b6d3e-1jammy ceph-common=19.1.1-298-gb53b6d3e-1jammy ceph-fuse=19.1.1-298-gb53b6d3e-1jammy ceph-test=19.1.1-298-gb53b6d3e-1jammy ceph-volume=19.1.1-298-gb53b6d3e-1jammy radosgw=19.1.1-298-gb53b6d3e-1jammy python3-rados=19.1.1-298-gb53b6d3e-1jammy python3-rgw=19.1.1-298-gb53b6d3e-1jammy python3-cephfs=19.1.1-298-gb53b6d3e-1jammy python3-rbd=19.1.1-298-gb53b6d3e-1jammy libcephfs2=19.1.1-298-gb53b6d3e-1jammy libcephfs-dev=19.1.1-298-gb53b6d3e-1jammy librados2=19.1.1-298-gb53b6d3e-1jammy librbd1=19.1.1-298-gb53b6d3e-1jammy rbd-fuse=19.1.1-298-gb53b6d3e-1jammy rbd-mirror=19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:36:28.550 INFO:teuthology.orchestra.run.smithi077.stdout:Reading package lists... 2024-09-11T08:36:28.627 INFO:teuthology.orchestra.run.smithi073.stdout:Building dependency tree... 2024-09-11T08:36:28.628 INFO:teuthology.orchestra.run.smithi073.stdout:Reading state information... 2024-09-11T08:36:28.742 INFO:teuthology.orchestra.run.smithi077.stdout:Building dependency tree... 2024-09-11T08:36:28.742 INFO:teuthology.orchestra.run.smithi077.stdout:Reading state information... 2024-09-11T08:36:28.823 INFO:teuthology.orchestra.run.smithi073.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T08:36:28.824 INFO:teuthology.orchestra.run.smithi073.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T08:36:28.824 INFO:teuthology.orchestra.run.smithi073.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T08:36:28.824 INFO:teuthology.orchestra.run.smithi073.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T08:36:28.824 INFO:teuthology.orchestra.run.smithi073.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T08:36:28.824 INFO:teuthology.orchestra.run.smithi073.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T08:36:28.825 INFO:teuthology.orchestra.run.smithi073.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T08:36:28.825 INFO:teuthology.orchestra.run.smithi073.stdout:The following additional packages will be installed: 2024-09-11T08:36:28.825 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T08:36:28.825 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T08:36:28.825 INFO:teuthology.orchestra.run.smithi073.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T08:36:28.826 INFO:teuthology.orchestra.run.smithi073.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T08:36:28.827 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: xmlstarlet zip 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout:Suggested packages: 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T08:36:28.828 INFO:teuthology.orchestra.run.smithi073.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T08:36:28.829 INFO:teuthology.orchestra.run.smithi073.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T08:36:28.829 INFO:teuthology.orchestra.run.smithi073.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T08:36:28.829 INFO:teuthology.orchestra.run.smithi073.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T08:36:28.829 INFO:teuthology.orchestra.run.smithi073.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T08:36:28.829 INFO:teuthology.orchestra.run.smithi073.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T08:36:28.829 INFO:teuthology.orchestra.run.smithi073.stdout:Recommended packages: 2024-09-11T08:36:28.829 INFO:teuthology.orchestra.run.smithi073.stdout: btrfs-tools 2024-09-11T08:36:28.927 INFO:teuthology.orchestra.run.smithi077.stdout:The following packages were automatically installed and are no longer required: 2024-09-11T08:36:28.927 INFO:teuthology.orchestra.run.smithi077.stdout: libboost-iostreams1.74.0 libboost-thread1.74.0 libgfapi0 libgfrpc0 libgfxdr0 2024-09-11T08:36:28.927 INFO:teuthology.orchestra.run.smithi077.stdout: libglusterfs0 libiscsi7 libpmemobj1 libpython2-dev libpython2-stdlib 2024-09-11T08:36:28.928 INFO:teuthology.orchestra.run.smithi077.stdout: libpython2.7 libpython2.7-dev libpython2.7-minimal libpython2.7-stdlib 2024-09-11T08:36:28.928 INFO:teuthology.orchestra.run.smithi077.stdout: linux-image-5.4.0-135-generic linux-modules-5.4.0-135-generic python2 2024-09-11T08:36:28.928 INFO:teuthology.orchestra.run.smithi077.stdout: python2-dev python2-minimal python2.7 python2.7-dev python2.7-minimal 2024-09-11T08:36:28.928 INFO:teuthology.orchestra.run.smithi077.stdout:Use 'sudo apt autoremove' to remove them. 2024-09-11T08:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout:The following additional packages will be installed: 2024-09-11T08:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-local 2024-09-11T08:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-k8sevents ceph-mgr-modules-core ceph-mon ceph-osd jq 2024-09-11T08:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout: libdouble-conversion3 libjq1 liblttng-ust1 liblua5.3-dev liboath0 libonig5 2024-09-11T08:36:28.929 INFO:teuthology.orchestra.run.smithi077.stdout: libpcre2-16-0 libqt5core5a libqt5dbus5 libqt5network5 libradosstriper1 2024-09-11T08:36:28.930 INFO:teuthology.orchestra.run.smithi077.stdout: librdkafka1 libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 2024-09-11T08:36:28.930 INFO:teuthology.orchestra.run.smithi077.stdout: lua-any lua-sec lua-socket lua5.1 luarocks nvme-cli pkg-config 2024-09-11T08:36:28.930 INFO:teuthology.orchestra.run.smithi077.stdout: python-asyncssh-doc python-babel-localedata python-pastedeploy-tpl 2024-09-11T08:36:28.930 INFO:teuthology.orchestra.run.smithi077.stdout: python3-asyncssh python3-babel python3-bcrypt python3-cachetools 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-argparse python3-ceph-common python3-certifi python3-cheroot 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pygments python3-pyinotify python3-pytest python3-repoze.lru 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-requests python3-requests-oauthlib python3-routes python3-rsa 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-simplegeneric python3-simplejson python3-singledispatch 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-sklearn python3-sklearn-lib python3-tempita python3-tempora 2024-09-11T08:36:28.931 INFO:teuthology.orchestra.run.smithi077.stdout: python3-threadpoolctl python3-toml python3-urllib3 python3-waitress 2024-09-11T08:36:28.932 INFO:teuthology.orchestra.run.smithi077.stdout: python3-wcwidth python3-webob python3-websocket python3-webtest 2024-09-11T08:36:28.932 INFO:teuthology.orchestra.run.smithi077.stdout: python3-werkzeug python3-zc.lockfile qttranslations5-l10n socat unzip 2024-09-11T08:36:28.932 INFO:teuthology.orchestra.run.smithi077.stdout: xmlstarlet zip 2024-09-11T08:36:28.932 INFO:teuthology.orchestra.run.smithi077.stdout:Suggested packages: 2024-09-11T08:36:28.932 INFO:teuthology.orchestra.run.smithi077.stdout: python3-influxdb readline-doc python-jinja2-doc python3-beaker 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout: python-mako-doc python-natsort-doc python-openssl-doc python3-openssl-dbg 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout: httpd-wsgi libapache2-mod-python libapache2-mod-scgi libjs-mochikit 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout: python-pecan-doc python-psutil-doc subversion python-pygments-doc 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout: python-pyinotify-doc python3-socks python-requests-doc python3-dap 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout: python-webtest-doc python-werkzeug-doc python3-watchdog 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout:Recommended packages: 2024-09-11T08:36:28.933 INFO:teuthology.orchestra.run.smithi077.stdout: btrfs-tools 2024-09-11T08:36:28.934 INFO:teuthology.orchestra.run.smithi073.stdout:The following NEW packages will be installed: 2024-09-11T08:36:28.934 INFO:teuthology.orchestra.run.smithi073.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T08:36:28.934 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T08:36:28.935 INFO:teuthology.orchestra.run.smithi073.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T08:36:28.936 INFO:teuthology.orchestra.run.smithi073.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-11T08:36:28.993 INFO:teuthology.orchestra.run.smithi073.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T08:36:28.994 INFO:teuthology.orchestra.run.smithi073.stdout:Need to get 184 MB of archives. 2024-09-11T08:36:28.994 INFO:teuthology.orchestra.run.smithi073.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-11T08:36:28.994 INFO:teuthology.orchestra.run.smithi073.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T08:36:29.038 INFO:teuthology.orchestra.run.smithi077.stdout:The following NEW packages will be installed: 2024-09-11T08:36:29.038 INFO:teuthology.orchestra.run.smithi077.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2024-09-11T08:36:29.038 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-local ceph-mgr-k8sevents 2024-09-11T08:36:29.038 INFO:teuthology.orchestra.run.smithi077.stdout: ceph-mgr-modules-core ceph-mon ceph-osd ceph-test ceph-volume cephadm jq 2024-09-11T08:36:29.038 INFO:teuthology.orchestra.run.smithi077.stdout: libcephfs-dev libcephfs2 libdouble-conversion3 libjq1 liblttng-ust1 2024-09-11T08:36:29.039 INFO:teuthology.orchestra.run.smithi077.stdout: liblua5.3-dev liboath0 libonig5 libpcre2-16-0 libqt5core5a libqt5dbus5 2024-09-11T08:36:29.039 INFO:teuthology.orchestra.run.smithi077.stdout: libqt5network5 librados2 libradosstriper1 librbd1 librdkafka1 2024-09-11T08:36:29.039 INFO:teuthology.orchestra.run.smithi077.stdout: libreadline-dev librgw2 libsqlite3-mod-ceph libthrift-0.16.0 lua-any lua-sec 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: lua-socket lua5.1 luarocks nvme-cli pkg-config python-asyncssh-doc 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python-babel-localedata python-pastedeploy-tpl python3-asyncssh 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-babel python3-bcrypt python3-cachetools python3-ceph-argparse 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-ceph-common python3-cephfs python3-certifi python3-cheroot 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-cherrypy3 python3-google-auth python3-idna python3-iniconfig 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.classes python3-jaraco.collections python3-jaraco.functools 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-jaraco.text python3-jinja2 python3-joblib python3-kubernetes 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-logutils python3-mako python3-markupsafe python3-natsort 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-openssl python3-paste python3-pastedeploy python3-pastescript 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pecan python3-pluggy python3-portend python3-prettytable 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-psutil python3-py python3-pyasn1 python3-pyasn1-modules 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-pygments python3-pyinotify python3-pytest python3-rados python3-rbd 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-repoze.lru python3-requests python3-requests-oauthlib python3-rgw 2024-09-11T08:36:29.040 INFO:teuthology.orchestra.run.smithi077.stdout: python3-routes python3-rsa python3-simplegeneric python3-simplejson 2024-09-11T08:36:29.041 INFO:teuthology.orchestra.run.smithi077.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-tempita 2024-09-11T08:36:29.041 INFO:teuthology.orchestra.run.smithi077.stdout: python3-tempora python3-threadpoolctl python3-toml python3-urllib3 2024-09-11T08:36:29.041 INFO:teuthology.orchestra.run.smithi077.stdout: python3-waitress python3-wcwidth python3-webob python3-websocket 2024-09-11T08:36:29.041 INFO:teuthology.orchestra.run.smithi077.stdout: python3-webtest python3-werkzeug python3-zc.lockfile qttranslations5-l10n 2024-09-11T08:36:29.041 INFO:teuthology.orchestra.run.smithi077.stdout: radosgw rbd-fuse rbd-mirror socat unzip xmlstarlet zip 2024-09-11T08:36:29.075 INFO:teuthology.orchestra.run.smithi073.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T08:36:29.082 INFO:teuthology.orchestra.run.smithi073.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T08:36:29.085 INFO:teuthology.orchestra.run.smithi073.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T08:36:29.098 INFO:teuthology.orchestra.run.smithi077.stdout:0 upgraded, 119 newly installed, 0 to remove and 331 not upgraded. 2024-09-11T08:36:29.098 INFO:teuthology.orchestra.run.smithi077.stdout:Need to get 184 MB of archives. 2024-09-11T08:36:29.098 INFO:teuthology.orchestra.run.smithi077.stdout:After this operation, 842 MB of additional disk space will be used. 2024-09-11T08:36:29.098 INFO:teuthology.orchestra.run.smithi077.stdout:Get:1 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblttng-ust1 amd64 2.13.1-1ubuntu1 [190 kB] 2024-09-11T08:36:29.102 INFO:teuthology.orchestra.run.smithi073.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T08:36:29.162 INFO:teuthology.orchestra.run.smithi073.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T08:36:29.167 INFO:teuthology.orchestra.run.smithi073.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T08:36:29.179 INFO:teuthology.orchestra.run.smithi073.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T08:36:29.182 INFO:teuthology.orchestra.run.smithi077.stdout:Get:2 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librados2 amd64 19.1.1-298-gb53b6d3e-1jammy [3,588 kB] 2024-09-11T08:36:29.183 INFO:teuthology.orchestra.run.smithi073.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T08:36:29.184 INFO:teuthology.orchestra.run.smithi073.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T08:36:29.185 INFO:teuthology.orchestra.run.smithi073.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T08:36:29.187 INFO:teuthology.orchestra.run.smithi077.stdout:Get:3 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libdouble-conversion3 amd64 3.1.7-4 [39.0 kB] 2024-09-11T08:36:29.189 INFO:teuthology.orchestra.run.smithi077.stdout:Get:4 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 libpcre2-16-0 amd64 10.39-3ubuntu0.1 [203 kB] 2024-09-11T08:36:29.193 INFO:teuthology.orchestra.run.smithi073.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T08:36:29.195 INFO:teuthology.orchestra.run.smithi073.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T08:36:29.198 INFO:teuthology.orchestra.run.smithi073.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T08:36:29.199 INFO:teuthology.orchestra.run.smithi073.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T08:36:29.201 INFO:teuthology.orchestra.run.smithi073.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T08:36:29.203 INFO:teuthology.orchestra.run.smithi073.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T08:36:29.206 INFO:teuthology.orchestra.run.smithi077.stdout:Get:5 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5core5a amd64 5.15.3+dfsg-2ubuntu0.2 [2,006 kB] 2024-09-11T08:36:29.217 INFO:teuthology.orchestra.run.smithi073.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T08:36:29.219 INFO:teuthology.orchestra.run.smithi073.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T08:36:29.219 INFO:teuthology.orchestra.run.smithi073.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T08:36:29.220 INFO:teuthology.orchestra.run.smithi073.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T08:36:29.221 INFO:teuthology.orchestra.run.smithi073.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T08:36:29.221 INFO:teuthology.orchestra.run.smithi073.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T08:36:29.221 INFO:teuthology.orchestra.run.smithi073.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T08:36:29.222 INFO:teuthology.orchestra.run.smithi073.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T08:36:29.236 INFO:teuthology.orchestra.run.smithi073.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T08:36:29.236 INFO:teuthology.orchestra.run.smithi073.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T08:36:29.253 INFO:teuthology.orchestra.run.smithi073.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T08:36:29.256 INFO:teuthology.orchestra.run.smithi073.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T08:36:29.256 INFO:teuthology.orchestra.run.smithi073.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T08:36:29.256 INFO:teuthology.orchestra.run.smithi073.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T08:36:29.257 INFO:teuthology.orchestra.run.smithi073.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T08:36:29.258 INFO:teuthology.orchestra.run.smithi073.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T08:36:29.258 INFO:teuthology.orchestra.run.smithi073.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T08:36:29.259 INFO:teuthology.orchestra.run.smithi073.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T08:36:29.259 INFO:teuthology.orchestra.run.smithi073.stdout:Get:36 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T08:36:29.267 INFO:teuthology.orchestra.run.smithi077.stdout:Get:6 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5dbus5 amd64 5.15.3+dfsg-2ubuntu0.2 [222 kB] 2024-09-11T08:36:29.270 INFO:teuthology.orchestra.run.smithi073.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T08:36:29.271 INFO:teuthology.orchestra.run.smithi073.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T08:36:29.272 INFO:teuthology.orchestra.run.smithi077.stdout:Get:7 http://archive.ubuntu.com/ubuntu jammy-updates/universe amd64 libqt5network5 amd64 5.15.3+dfsg-2ubuntu0.2 [731 kB] 2024-09-11T08:36:29.283 INFO:teuthology.orchestra.run.smithi077.stdout:Get:8 http://archive.ubuntu.com/ubuntu jammy/universe amd64 libthrift-0.16.0 amd64 0.16.0-2 [267 kB] 2024-09-11T08:36:29.287 INFO:teuthology.orchestra.run.smithi073.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T08:36:29.287 INFO:teuthology.orchestra.run.smithi077.stdout:Get:9 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-wcwidth all 0.2.5+dfsg1-1 [21.9 kB] 2024-09-11T08:36:29.288 INFO:teuthology.orchestra.run.smithi077.stdout:Get:10 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-prettytable all 2.5.0-2 [31.3 kB] 2024-09-11T08:36:29.288 INFO:teuthology.orchestra.run.smithi077.stdout:Get:11 http://archive.ubuntu.com/ubuntu jammy/universe amd64 librdkafka1 amd64 1.8.0-1build1 [633 kB] 2024-09-11T08:36:29.292 INFO:teuthology.orchestra.run.smithi073.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T08:36:29.297 INFO:teuthology.orchestra.run.smithi073.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T08:36:29.298 INFO:teuthology.orchestra.run.smithi077.stdout:Get:12 http://archive.ubuntu.com/ubuntu jammy/main amd64 libreadline-dev amd64 8.1.2-1 [166 kB] 2024-09-11T08:36:29.298 INFO:teuthology.orchestra.run.smithi073.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T08:36:29.298 INFO:teuthology.orchestra.run.smithi073.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T08:36:29.298 INFO:teuthology.orchestra.run.smithi073.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T08:36:29.298 INFO:teuthology.orchestra.run.smithi073.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T08:36:29.298 INFO:teuthology.orchestra.run.smithi073.stdout:Get:46 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T08:36:29.299 INFO:teuthology.orchestra.run.smithi077.stdout:Get:13 http://archive.ubuntu.com/ubuntu jammy/main amd64 liblua5.3-dev amd64 5.3.6-1build1 [167 kB] 2024-09-11T08:36:29.301 INFO:teuthology.orchestra.run.smithi077.stdout:Get:14 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua5.1 amd64 5.1.5-8.1build4 [94.6 kB] 2024-09-11T08:36:29.302 INFO:teuthology.orchestra.run.smithi077.stdout:Get:15 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-any all 27ubuntu1 [5,034 B] 2024-09-11T08:36:29.304 INFO:teuthology.orchestra.run.smithi073.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T08:36:29.305 INFO:teuthology.orchestra.run.smithi073.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T08:36:29.305 INFO:teuthology.orchestra.run.smithi077.stdout:Get:16 http://archive.ubuntu.com/ubuntu jammy/main amd64 zip amd64 3.0-12build2 [176 kB] 2024-09-11T08:36:29.307 INFO:teuthology.orchestra.run.smithi077.stdout:Get:17 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 unzip amd64 6.0-26ubuntu3.2 [175 kB] 2024-09-11T08:36:29.309 INFO:teuthology.orchestra.run.smithi077.stdout:Get:18 http://archive.ubuntu.com/ubuntu jammy/universe amd64 luarocks all 3.8.0+dfsg1-1 [140 kB] 2024-09-11T08:36:29.321 INFO:teuthology.orchestra.run.smithi073.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T08:36:29.322 INFO:teuthology.orchestra.run.smithi073.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T08:36:29.322 INFO:teuthology.orchestra.run.smithi077.stdout:Get:19 http://archive.ubuntu.com/ubuntu jammy/main amd64 liboath0 amd64 2.6.7-3build1 [41.1 kB] 2024-09-11T08:36:29.322 INFO:teuthology.orchestra.run.smithi073.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T08:36:29.323 INFO:teuthology.orchestra.run.smithi077.stdout:Get:20 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.functools all 3.4.0-2 [9,030 B] 2024-09-11T08:36:29.323 INFO:teuthology.orchestra.run.smithi077.stdout:Get:21 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-cheroot all 8.5.2+ds1-1ubuntu3.1 [71.1 kB] 2024-09-11T08:36:29.324 INFO:teuthology.orchestra.run.smithi077.stdout:Get:22 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.classes all 3.2.1-3 [6,452 B] 2024-09-11T08:36:29.324 INFO:teuthology.orchestra.run.smithi077.stdout:Get:23 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.text all 3.6.0-2 [8,716 B] 2024-09-11T08:36:29.325 INFO:teuthology.orchestra.run.smithi073.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T08:36:29.325 INFO:teuthology.orchestra.run.smithi077.stdout:Get:24 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-jaraco.collections all 3.4.0-2 [11.4 kB] 2024-09-11T08:36:29.325 INFO:teuthology.orchestra.run.smithi077.stdout:Get:25 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempora all 4.1.2-1 [14.8 kB] 2024-09-11T08:36:29.338 INFO:teuthology.orchestra.run.smithi073.stdout:Get:53 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T08:36:29.339 INFO:teuthology.orchestra.run.smithi077.stdout:Get:26 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-portend all 3.0.0-1 [7,240 B] 2024-09-11T08:36:29.339 INFO:teuthology.orchestra.run.smithi077.stdout:Get:27 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-zc.lockfile all 2.0-1 [8,980 B] 2024-09-11T08:36:29.339 INFO:teuthology.orchestra.run.smithi077.stdout:Get:28 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cherrypy3 all 18.6.1-4 [208 kB] 2024-09-11T08:36:29.353 INFO:teuthology.orchestra.run.smithi073.stdout:Get:54 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T08:36:29.355 INFO:teuthology.orchestra.run.smithi077.stdout:Get:29 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-natsort all 8.0.2-1 [35.3 kB] 2024-09-11T08:36:29.356 INFO:teuthology.orchestra.run.smithi077.stdout:Get:30 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-logutils all 0.3.3-8 [17.6 kB] 2024-09-11T08:36:29.357 INFO:teuthology.orchestra.run.smithi077.stdout:Get:31 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-markupsafe amd64 2.0.1-2build1 [12.7 kB] 2024-09-11T08:36:29.357 INFO:teuthology.orchestra.run.smithi077.stdout:Get:32 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-mako all 1.1.3+ds1-2ubuntu0.1 [60.5 kB] 2024-09-11T08:36:29.358 INFO:teuthology.orchestra.run.smithi077.stdout:Get:33 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2024-09-11T08:36:29.358 INFO:teuthology.orchestra.run.smithi077.stdout:Get:34 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-singledispatch all 3.4.0.3-3 [7,320 B] 2024-09-11T08:36:29.359 INFO:teuthology.orchestra.run.smithi077.stdout:Get:35 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-webob all 1:1.8.6-1.1ubuntu0.1 [86.7 kB] 2024-09-11T08:36:29.359 INFO:teuthology.orchestra.run.smithi073.stdout:Get:55 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T08:36:29.360 INFO:teuthology.orchestra.run.smithi073.stdout:Get:56 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T08:36:29.372 INFO:teuthology.orchestra.run.smithi077.stdout:Get:36 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-waitress all 1.4.4-1.1ubuntu1 [47.3 kB] 2024-09-11T08:36:29.373 INFO:teuthology.orchestra.run.smithi077.stdout:Get:37 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-tempita all 0.5.2-6ubuntu1 [15.1 kB] 2024-09-11T08:36:29.373 INFO:teuthology.orchestra.run.smithi077.stdout:Get:38 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-paste all 3.5.0+dfsg1-1 [456 kB] 2024-09-11T08:36:29.376 INFO:teuthology.orchestra.run.smithi073.stdout:Get:57 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T08:36:29.377 INFO:teuthology.orchestra.run.smithi073.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T08:36:29.378 INFO:teuthology.orchestra.run.smithi073.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T08:36:29.378 INFO:teuthology.orchestra.run.smithi073.stdout:Get:60 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T08:36:29.379 INFO:teuthology.orchestra.run.smithi073.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T08:36:29.389 INFO:teuthology.orchestra.run.smithi077.stdout:Get:39 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-pastedeploy-tpl all 2.1.1-1 [4,892 B] 2024-09-11T08:36:29.390 INFO:teuthology.orchestra.run.smithi077.stdout:Get:40 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastedeploy all 2.1.1-1 [26.6 kB] 2024-09-11T08:36:29.390 INFO:teuthology.orchestra.run.smithi077.stdout:Get:41 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-webtest all 2.0.35-1 [28.5 kB] 2024-09-11T08:36:29.391 INFO:teuthology.orchestra.run.smithi073.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T08:36:29.391 INFO:teuthology.orchestra.run.smithi073.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T08:36:29.392 INFO:teuthology.orchestra.run.smithi077.stdout:Get:42 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pecan all 1.3.3-4ubuntu2 [87.3 kB] 2024-09-11T08:36:29.392 INFO:teuthology.orchestra.run.smithi073.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T08:36:29.392 INFO:teuthology.orchestra.run.smithi077.stdout:Get:43 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-certifi all 2020.6.20-1 [150 kB] 2024-09-11T08:36:29.394 INFO:teuthology.orchestra.run.smithi077.stdout:Get:44 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-idna all 3.3-1ubuntu0.1 [52.1 kB] 2024-09-11T08:36:29.394 INFO:teuthology.orchestra.run.smithi077.stdout:Get:45 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-urllib3 all 1.26.5-1~exp1ubuntu0.1 [98.2 kB] 2024-09-11T08:36:29.400 INFO:teuthology.orchestra.run.smithi077.stdout:Get:46 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librbd1 amd64 19.1.1-298-gb53b6d3e-1jammy [3,243 kB] 2024-09-11T08:36:29.406 INFO:teuthology.orchestra.run.smithi077.stdout:Get:47 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-requests all 2.25.1+dfsg-2ubuntu0.1 [48.8 kB] 2024-09-11T08:36:29.407 INFO:teuthology.orchestra.run.smithi077.stdout:Get:48 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-werkzeug all 2.0.2+dfsg1-1ubuntu0.22.04.2 [181 kB] 2024-09-11T08:36:29.409 INFO:teuthology.orchestra.run.smithi077.stdout:Get:49 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-bcrypt amd64 3.2.0-1build1 [32.7 kB] 2024-09-11T08:36:29.409 INFO:teuthology.orchestra.run.smithi073.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T08:36:29.412 INFO:teuthology.orchestra.run.smithi073.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T08:36:29.412 INFO:teuthology.orchestra.run.smithi073.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T08:36:29.422 INFO:teuthology.orchestra.run.smithi077.stdout:Get:50 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-openssl all 21.0.0-1 [45.2 kB] 2024-09-11T08:36:29.423 INFO:teuthology.orchestra.run.smithi077.stdout:Get:51 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-asyncssh all 2.5.0-1 [185 kB] 2024-09-11T08:36:29.426 INFO:teuthology.orchestra.run.smithi077.stdout:Get:52 http://archive.ubuntu.com/ubuntu jammy/main amd64 python-babel-localedata all 2.8.0+dfsg.1-7 [4,982 kB] 2024-09-11T08:36:29.428 INFO:teuthology.orchestra.run.smithi073.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T08:36:29.428 INFO:teuthology.orchestra.run.smithi073.stdout:Get:69 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T08:36:29.429 INFO:teuthology.orchestra.run.smithi073.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T08:36:29.429 INFO:teuthology.orchestra.run.smithi073.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T08:36:29.429 INFO:teuthology.orchestra.run.smithi073.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T08:36:29.430 INFO:teuthology.orchestra.run.smithi073.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T08:36:29.430 INFO:teuthology.orchestra.run.smithi073.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T08:36:29.431 INFO:teuthology.orchestra.run.smithi073.stdout:Get:75 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T08:36:29.448 INFO:teuthology.orchestra.run.smithi073.stdout:Get:76 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T08:36:29.450 INFO:teuthology.orchestra.run.smithi073.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T08:36:29.451 INFO:teuthology.orchestra.run.smithi073.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T08:36:29.451 INFO:teuthology.orchestra.run.smithi073.stdout:Get:79 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T08:36:29.455 INFO:teuthology.orchestra.run.smithi073.stdout:Get:80 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T08:36:29.457 INFO:teuthology.orchestra.run.smithi073.stdout:Get:81 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T08:36:29.457 INFO:teuthology.orchestra.run.smithi073.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T08:36:29.458 INFO:teuthology.orchestra.run.smithi073.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T08:36:29.465 INFO:teuthology.orchestra.run.smithi073.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T08:36:29.465 INFO:teuthology.orchestra.run.smithi073.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T08:36:29.481 INFO:teuthology.orchestra.run.smithi077.stdout:Get:53 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-babel all 2.8.0+dfsg.1-7 [85.1 kB] 2024-09-11T08:36:29.481 INFO:teuthology.orchestra.run.smithi077.stdout:Get:54 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 python3-jinja2 all 3.0.3-1ubuntu0.2 [108 kB] 2024-09-11T08:36:29.482 INFO:teuthology.orchestra.run.smithi073.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T08:36:29.482 INFO:teuthology.orchestra.run.smithi073.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T08:36:29.482 INFO:teuthology.orchestra.run.smithi073.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T08:36:29.483 INFO:teuthology.orchestra.run.smithi073.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T08:36:29.483 INFO:teuthology.orchestra.run.smithi077.stdout:Get:55 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2024-09-11T08:36:29.483 INFO:teuthology.orchestra.run.smithi077.stdout:Get:56 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-routes all 2.5.1-1ubuntu1 [89.0 kB] 2024-09-11T08:36:29.484 INFO:teuthology.orchestra.run.smithi077.stdout:Get:57 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn-lib amd64 0.23.2-5ubuntu6 [2,058 kB] 2024-09-11T08:36:29.484 INFO:teuthology.orchestra.run.smithi073.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T08:36:29.485 INFO:teuthology.orchestra.run.smithi073.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T08:36:29.491 INFO:teuthology.orchestra.run.smithi073.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T08:36:29.491 INFO:teuthology.orchestra.run.smithi073.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T08:36:29.500 INFO:teuthology.orchestra.run.smithi073.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T08:36:29.501 INFO:teuthology.orchestra.run.smithi073.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T08:36:29.502 INFO:teuthology.orchestra.run.smithi077.stdout:Get:58 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-joblib all 0.17.0-4ubuntu1 [204 kB] 2024-09-11T08:36:29.504 INFO:teuthology.orchestra.run.smithi077.stdout:Get:59 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-threadpoolctl all 3.1.0-1 [21.3 kB] 2024-09-11T08:36:29.504 INFO:teuthology.orchestra.run.smithi077.stdout:Get:60 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-sklearn all 0.23.2-5ubuntu6 [1,829 kB] 2024-09-11T08:36:29.516 INFO:teuthology.orchestra.run.smithi073.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T08:36:29.519 INFO:teuthology.orchestra.run.smithi077.stdout:Get:61 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-cachetools all 5.0.0-1 [9,722 B] 2024-09-11T08:36:29.519 INFO:teuthology.orchestra.run.smithi077.stdout:Get:62 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1 all 0.4.8-1 [50.9 kB] 2024-09-11T08:36:29.522 INFO:teuthology.orchestra.run.smithi077.stdout:Get:63 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs2 amd64 19.1.1-298-gb53b6d3e-1jammy [719 kB] 2024-09-11T08:36:29.523 INFO:teuthology.orchestra.run.smithi077.stdout:Get:64 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyasn1-modules all 0.2.1-1 [33.0 kB] 2024-09-11T08:36:29.523 INFO:teuthology.orchestra.run.smithi077.stdout:Get:65 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-rsa all 4.8-1 [28.4 kB] 2024-09-11T08:36:29.524 INFO:teuthology.orchestra.run.smithi077.stdout:Get:66 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-google-auth all 1.5.1-3 [35.7 kB] 2024-09-11T08:36:29.524 INFO:teuthology.orchestra.run.smithi077.stdout:Get:67 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-requests-oauthlib all 1.3.0+ds-0.1 [18.7 kB] 2024-09-11T08:36:29.553 INFO:teuthology.orchestra.run.smithi077.stdout:Get:68 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-websocket all 1.2.3-1 [34.7 kB] 2024-09-11T08:36:29.553 INFO:teuthology.orchestra.run.smithi077.stdout:Get:69 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rados amd64 19.1.1-298-gb53b6d3e-1jammy [356 kB] 2024-09-11T08:36:29.553 INFO:teuthology.orchestra.run.smithi077.stdout:Get:70 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-kubernetes all 12.0.1-1ubuntu1 [353 kB] 2024-09-11T08:36:29.556 INFO:teuthology.orchestra.run.smithi077.stdout:Get:71 http://archive.ubuntu.com/ubuntu jammy/main amd64 libonig5 amd64 6.9.7.1-2build1 [172 kB] 2024-09-11T08:36:29.557 INFO:teuthology.orchestra.run.smithi077.stdout:Get:72 http://archive.ubuntu.com/ubuntu jammy/main amd64 libjq1 amd64 1.6-2.1ubuntu3 [133 kB] 2024-09-11T08:36:29.559 INFO:teuthology.orchestra.run.smithi077.stdout:Get:73 http://archive.ubuntu.com/ubuntu jammy/main amd64 jq amd64 1.6-2.1ubuntu3 [52.5 kB] 2024-09-11T08:36:29.559 INFO:teuthology.orchestra.run.smithi077.stdout:Get:74 http://archive.ubuntu.com/ubuntu jammy/main amd64 socat amd64 1.7.4.1-3ubuntu4 [349 kB] 2024-09-11T08:36:29.560 INFO:teuthology.orchestra.run.smithi077.stdout:Get:75 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-argparse all 19.1.1-298-gb53b6d3e-1jammy [32.8 kB] 2024-09-11T08:36:29.561 INFO:teuthology.orchestra.run.smithi077.stdout:Get:76 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-cephfs amd64 19.1.1-298-gb53b6d3e-1jammy [180 kB] 2024-09-11T08:36:29.562 INFO:teuthology.orchestra.run.smithi077.stdout:Get:77 http://archive.ubuntu.com/ubuntu jammy/universe amd64 xmlstarlet amd64 1.6.1-2.1 [265 kB] 2024-09-11T08:36:29.564 INFO:teuthology.orchestra.run.smithi077.stdout:Get:78 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-socket amd64 3.0~rc1+git+ac3201d-6 [78.9 kB] 2024-09-11T08:36:29.569 INFO:teuthology.orchestra.run.smithi077.stdout:Get:79 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-ceph-common all 19.1.1-298-gb53b6d3e-1jammy [67.6 kB] 2024-09-11T08:36:29.571 INFO:teuthology.orchestra.run.smithi077.stdout:Get:80 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rbd amd64 19.1.1-298-gb53b6d3e-1jammy [333 kB] 2024-09-11T08:36:29.578 INFO:teuthology.orchestra.run.smithi077.stdout:Get:81 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 librgw2 amd64 19.1.1-298-gb53b6d3e-1jammy [6,875 kB] 2024-09-11T08:36:29.583 INFO:teuthology.orchestra.run.smithi077.stdout:Get:82 http://archive.ubuntu.com/ubuntu jammy/universe amd64 lua-sec amd64 1.0.2-1 [37.6 kB] 2024-09-11T08:36:29.584 INFO:teuthology.orchestra.run.smithi077.stdout:Get:83 http://archive.ubuntu.com/ubuntu jammy-updates/main amd64 nvme-cli amd64 1.16-3ubuntu0.2 [473 kB] 2024-09-11T08:36:29.584 INFO:teuthology.orchestra.run.smithi073.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T08:36:29.586 INFO:teuthology.orchestra.run.smithi073.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T08:36:29.588 INFO:teuthology.orchestra.run.smithi077.stdout:Get:84 http://archive.ubuntu.com/ubuntu jammy/main amd64 pkg-config amd64 0.29.2-1ubuntu3 [48.2 kB] 2024-09-11T08:36:29.594 INFO:teuthology.orchestra.run.smithi073.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T08:36:29.600 INFO:teuthology.orchestra.run.smithi077.stdout:Get:85 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python-asyncssh-doc all 2.5.0-1 [318 kB] 2024-09-11T08:36:29.603 INFO:teuthology.orchestra.run.smithi077.stdout:Get:86 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-iniconfig all 1.1.1-2 [6,024 B] 2024-09-11T08:36:29.603 INFO:teuthology.orchestra.run.smithi077.stdout:Get:87 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2024-09-11T08:36:29.603 INFO:teuthology.orchestra.run.smithi077.stdout:Get:88 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pluggy all 0.13.0-7.1 [19.0 kB] 2024-09-11T08:36:29.604 INFO:teuthology.orchestra.run.smithi077.stdout:Get:89 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-psutil amd64 5.9.0-1build1 [158 kB] 2024-09-11T08:36:29.605 INFO:teuthology.orchestra.run.smithi077.stdout:Get:90 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-py all 1.10.0-1 [71.9 kB] 2024-09-11T08:36:29.606 INFO:teuthology.orchestra.run.smithi077.stdout:Get:91 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pygments all 2.11.2+dfsg-2 [750 kB] 2024-09-11T08:36:29.617 INFO:teuthology.orchestra.run.smithi077.stdout:Get:92 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-pyinotify all 0.9.6-1.3 [24.8 kB] 2024-09-11T08:36:29.617 INFO:teuthology.orchestra.run.smithi077.stdout:Get:93 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-toml all 0.10.2-1 [16.5 kB] 2024-09-11T08:36:29.617 INFO:teuthology.orchestra.run.smithi077.stdout:Get:94 http://archive.ubuntu.com/ubuntu jammy/universe amd64 python3-pytest all 6.2.5-1ubuntu2 [214 kB] 2024-09-11T08:36:29.634 INFO:teuthology.orchestra.run.smithi077.stdout:Get:95 http://archive.ubuntu.com/ubuntu jammy/main amd64 python3-simplejson amd64 3.17.6-1build1 [54.7 kB] 2024-09-11T08:36:29.634 INFO:teuthology.orchestra.run.smithi077.stdout:Get:96 http://archive.ubuntu.com/ubuntu jammy/universe amd64 qttranslations5-l10n all 5.15.3-1 [1,983 kB] 2024-09-11T08:36:29.838 INFO:teuthology.orchestra.run.smithi077.stdout:Get:97 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 python3-rgw amd64 19.1.1-298-gb53b6d3e-1jammy [112 kB] 2024-09-11T08:36:29.841 INFO:teuthology.orchestra.run.smithi077.stdout:Get:98 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libradosstriper1 amd64 19.1.1-298-gb53b6d3e-1jammy [472 kB] 2024-09-11T08:36:29.861 INFO:teuthology.orchestra.run.smithi077.stdout:Get:99 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-common amd64 19.1.1-298-gb53b6d3e-1jammy [25.9 MB] 2024-09-11T08:36:30.509 INFO:teuthology.orchestra.run.smithi073.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T08:36:30.586 INFO:teuthology.orchestra.run.smithi073.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T08:36:30.589 INFO:teuthology.orchestra.run.smithi073.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T08:36:30.590 INFO:teuthology.orchestra.run.smithi073.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T08:36:30.616 INFO:teuthology.orchestra.run.smithi073.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T08:36:30.796 INFO:teuthology.orchestra.run.smithi073.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T08:36:30.850 INFO:teuthology.orchestra.run.smithi077.stdout:Get:100 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-base amd64 19.1.1-298-gb53b6d3e-1jammy [5,145 kB] 2024-09-11T08:36:30.975 INFO:teuthology.orchestra.run.smithi077.stdout:Get:101 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-modules-core all 19.1.1-298-gb53b6d3e-1jammy [245 kB] 2024-09-11T08:36:30.996 INFO:teuthology.orchestra.run.smithi077.stdout:Get:102 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libsqlite3-mod-ceph amd64 19.1.1-298-gb53b6d3e-1jammy [124 kB] 2024-09-11T08:36:31.000 INFO:teuthology.orchestra.run.smithi077.stdout:Get:103 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr amd64 19.1.1-298-gb53b6d3e-1jammy [1,555 kB] 2024-09-11T08:36:31.054 INFO:teuthology.orchestra.run.smithi077.stdout:Get:104 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mon amd64 19.1.1-298-gb53b6d3e-1jammy [6,143 kB] 2024-09-11T08:36:31.282 INFO:teuthology.orchestra.run.smithi077.stdout:Get:105 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-osd amd64 19.1.1-298-gb53b6d3e-1jammy [22.8 MB] 2024-09-11T08:36:31.532 INFO:teuthology.orchestra.run.smithi073.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T08:36:31.579 INFO:teuthology.orchestra.run.smithi073.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T08:36:31.596 INFO:teuthology.orchestra.run.smithi073.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T08:36:31.638 INFO:teuthology.orchestra.run.smithi073.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T08:36:31.648 INFO:teuthology.orchestra.run.smithi073.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T08:36:31.650 INFO:teuthology.orchestra.run.smithi073.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T08:36:31.686 INFO:teuthology.orchestra.run.smithi073.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T08:36:31.941 INFO:teuthology.orchestra.run.smithi073.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T08:36:31.941 INFO:teuthology.orchestra.run.smithi073.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T08:36:32.177 INFO:teuthology.orchestra.run.smithi077.stdout:Get:106 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph amd64 19.1.1-298-gb53b6d3e-1jammy [14.1 kB] 2024-09-11T08:36:32.178 INFO:teuthology.orchestra.run.smithi077.stdout:Get:107 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [877 kB] 2024-09-11T08:36:32.195 INFO:teuthology.orchestra.run.smithi077.stdout:Get:108 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mds amd64 19.1.1-298-gb53b6d3e-1jammy [2,462 kB] 2024-09-11T08:36:32.236 INFO:teuthology.orchestra.run.smithi077.stdout:Get:109 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 cephadm amd64 19.1.1-298-gb53b6d3e-1jammy [794 kB] 2024-09-11T08:36:32.246 INFO:teuthology.orchestra.run.smithi077.stdout:Get:110 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-cephadm all 19.1.1-298-gb53b6d3e-1jammy [152 kB] 2024-09-11T08:36:32.268 INFO:teuthology.orchestra.run.smithi077.stdout:Get:111 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-dashboard all 19.1.1-298-gb53b6d3e-1jammy [2,284 kB] 2024-09-11T08:36:32.338 INFO:teuthology.orchestra.run.smithi077.stdout:Get:112 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-diskprediction-local all 19.1.1-298-gb53b6d3e-1jammy [8,625 kB] 2024-09-11T08:36:32.666 INFO:teuthology.orchestra.run.smithi077.stdout:Get:113 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-mgr-k8sevents all 19.1.1-298-gb53b6d3e-1jammy [14.2 kB] 2024-09-11T08:36:32.666 INFO:teuthology.orchestra.run.smithi077.stdout:Get:114 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-test amd64 19.1.1-298-gb53b6d3e-1jammy [51.4 MB] 2024-09-11T08:36:33.684 INFO:teuthology.orchestra.run.smithi073.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T08:36:33.686 INFO:teuthology.orchestra.run.smithi073.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T08:36:33.687 INFO:teuthology.orchestra.run.smithi073.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T08:36:34.018 INFO:teuthology.orchestra.run.smithi073.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T08:36:34.020 INFO:teuthology.orchestra.run.smithi073.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-298-gb53b6d3e-1jammy [3,140 kB] 2024-09-11T08:36:34.479 INFO:teuthology.orchestra.run.smithi077.stdout:Get:115 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 ceph-volume all 19.1.1-298-gb53b6d3e-1jammy [130 kB] 2024-09-11T08:36:34.482 INFO:teuthology.orchestra.run.smithi077.stdout:Get:116 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 libcephfs-dev amd64 19.1.1-298-gb53b6d3e-1jammy [39.0 kB] 2024-09-11T08:36:34.483 INFO:teuthology.orchestra.run.smithi077.stdout:Get:117 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 radosgw amd64 19.1.1-298-gb53b6d3e-1jammy [13.6 MB] 2024-09-11T08:36:34.673 INFO:teuthology.orchestra.run.smithi077.stdout:Get:118 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-fuse amd64 19.1.1-298-gb53b6d3e-1jammy [91.5 kB] 2024-09-11T08:36:34.674 INFO:teuthology.orchestra.run.smithi077.stdout:Get:119 https://1.chacra.ceph.com/r/ceph/squid/b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc/ubuntu/jammy/flavors/default jammy/main amd64 rbd-mirror amd64 19.1.1-298-gb53b6d3e-1jammy [3,140 kB] 2024-09-11T08:36:34.693 INFO:teuthology.orchestra.run.smithi073.stdout:Fetched 184 MB in 5s (35.6 MB/s) 2024-09-11T08:36:34.822 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T08:36:34.883 INFO:teuthology.orchestra.run.smithi073.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T08:36:34.888 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T08:36:34.955 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T08:36:35.292 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T08:36:35.298 INFO:teuthology.orchestra.run.smithi077.stdout:Fetched 184 MB in 6s (32.4 MB/s) 2024-09-11T08:36:35.318 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T08:36:35.350 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T08:36:35.451 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liblttng-ust1:amd64. 2024-09-11T08:36:35.548 INFO:teuthology.orchestra.run.smithi077.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 165988 files and directories currently installed.) 2024-09-11T08:36:35.553 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../000-liblttng-ust1_2.13.1-1ubuntu1_amd64.deb ... 2024-09-11T08:36:35.641 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T08:36:35.678 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T08:36:35.704 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T08:36:35.744 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T08:36:35.978 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libdouble-conversion3:amd64. 2024-09-11T08:36:36.004 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../001-libdouble-conversion3_3.1.7-4_amd64.deb ... 2024-09-11T08:36:36.044 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T08:36:36.182 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T08:36:36.208 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T08:36:36.256 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:36:36.414 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libpcre2-16-0:amd64. 2024-09-11T08:36:36.441 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../002-libpcre2-16-0_10.39-3ubuntu0.1_amd64.deb ... 2024-09-11T08:36:36.481 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T08:36:36.618 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T08:36:36.644 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T08:36:36.685 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:36:36.901 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libqt5core5a:amd64. 2024-09-11T08:36:36.927 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../003-libqt5core5a_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T08:36:36.975 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:36:37.021 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T08:36:37.048 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T08:36:37.088 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:36:37.370 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libqt5dbus5:amd64. 2024-09-11T08:36:37.396 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../004-libqt5dbus5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T08:36:37.437 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:36:37.466 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T08:36:37.492 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T08:36:37.533 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T08:36:37.773 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libqt5network5:amd64. 2024-09-11T08:36:37.800 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../005-libqt5network5_5.15.3+dfsg-2ubuntu0.2_amd64.deb ... 2024-09-11T08:36:37.811 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package librados2. 2024-09-11T08:36:37.837 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:37.839 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:36:37.877 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:38.142 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libthrift-0.16.0:amd64. 2024-09-11T08:36:38.158 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../006-libthrift-0.16.0_0.16.0-2_amd64.deb ... 2024-09-11T08:36:38.192 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T08:36:38.347 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package librbd1. 2024-09-11T08:36:38.373 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:38.414 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:38.512 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librados2. 2024-09-11T08:36:38.538 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../007-librados2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:38.578 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:38.875 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T08:36:38.901 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:38.941 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:39.131 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librbd1. 2024-09-11T08:36:39.158 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../008-librbd1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:39.198 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:39.219 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-rados. 2024-09-11T08:36:39.245 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:39.285 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:39.588 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T08:36:39.614 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:36:39.633 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libcephfs2. 2024-09-11T08:36:39.657 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../009-libcephfs2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:39.663 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:39.692 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:39.933 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T08:36:39.944 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rados. 2024-09-11T08:36:39.947 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:39.971 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../010-python3-rados_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:39.991 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:40.011 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:40.302 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T08:36:40.328 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:36:40.355 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-ceph-argparse. 2024-09-11T08:36:40.368 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:40.381 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../011-python3-ceph-argparse_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:36:40.421 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:40.674 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cephfs. 2024-09-11T08:36:40.679 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T08:36:40.690 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../012-python3-cephfs_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:40.695 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T08:36:40.732 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:40.747 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T08:36:41.027 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-ceph-common. 2024-09-11T08:36:41.041 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T08:36:41.053 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../013-python3-ceph-common_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:36:41.067 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T08:36:41.093 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:41.108 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T08:36:41.369 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T08:36:41.393 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:41.412 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-wcwidth. 2024-09-11T08:36:41.435 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:41.439 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../014-python3-wcwidth_0.2.5+dfsg1-1_all.deb ... 2024-09-11T08:36:41.471 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T08:36:41.781 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T08:36:41.782 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-prettytable. 2024-09-11T08:36:41.806 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T08:36:41.808 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../015-python3-prettytable_2.5.0-2_all.deb ... 2024-09-11T08:36:41.847 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T08:36:41.848 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-prettytable (2.5.0-2) ... 2024-09-11T08:36:42.101 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rbd. 2024-09-11T08:36:42.116 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T08:36:42.116 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../016-python3-rbd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:42.131 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T08:36:42.159 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:42.166 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T08:36:42.469 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T08:36:42.484 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T08:36:42.527 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T08:36:42.695 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librdkafka1:amd64. 2024-09-11T08:36:42.722 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../017-librdkafka1_1.8.0-1build1_amd64.deb ... 2024-09-11T08:36:42.762 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T08:36:42.864 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package lua5.1. 2024-09-11T08:36:42.890 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T08:36:42.931 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T08:36:43.031 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libreadline-dev:amd64. 2024-09-11T08:36:43.047 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../018-libreadline-dev_8.1.2-1_amd64.deb ... 2024-09-11T08:36:43.081 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T08:36:43.242 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package lua-any. 2024-09-11T08:36:43.257 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T08:36:43.300 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T08:36:43.334 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liblua5.3-dev:amd64. 2024-09-11T08:36:43.349 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../019-liblua5.3-dev_5.3.6-1build1_amd64.deb ... 2024-09-11T08:36:43.384 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T08:36:43.545 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package zip. 2024-09-11T08:36:43.571 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T08:36:43.611 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T08:36:43.687 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua5.1. 2024-09-11T08:36:43.713 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../020-lua5.1_5.1.5-8.1build4_amd64.deb ... 2024-09-11T08:36:43.753 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua5.1 (5.1.5-8.1build4) ... 2024-09-11T08:36:43.898 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package unzip. 2024-09-11T08:36:43.924 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T08:36:43.964 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T08:36:44.090 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua-any. 2024-09-11T08:36:44.105 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../021-lua-any_27ubuntu1_all.deb ... 2024-09-11T08:36:44.148 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua-any (27ubuntu1) ... 2024-09-11T08:36:44.368 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package luarocks. 2024-09-11T08:36:44.375 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package zip. 2024-09-11T08:36:44.391 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../022-zip_3.0-12build2_amd64.deb ... 2024-09-11T08:36:44.395 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T08:36:44.425 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking zip (3.0-12build2) ... 2024-09-11T08:36:44.435 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T08:36:44.737 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package unzip. 2024-09-11T08:36:44.763 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../023-unzip_6.0-26ubuntu3.2_amd64.deb ... 2024-09-11T08:36:44.801 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package librgw2. 2024-09-11T08:36:44.803 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking unzip (6.0-26ubuntu3.2) ... 2024-09-11T08:36:44.828 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:44.868 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:45.184 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package luarocks. 2024-09-11T08:36:45.210 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../024-luarocks_3.8.0+dfsg1-1_all.deb ... 2024-09-11T08:36:45.250 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking luarocks (3.8.0+dfsg1-1) ... 2024-09-11T08:36:45.379 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T08:36:45.394 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:45.428 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:45.674 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package librgw2. 2024-09-11T08:36:45.701 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../025-librgw2_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:45.733 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:45.765 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T08:36:45.791 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T08:36:45.832 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T08:36:46.118 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T08:36:46.140 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:46.184 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:46.252 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rgw. 2024-09-11T08:36:46.267 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../026-python3-rgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:46.310 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:46.546 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-common. 2024-09-11T08:36:46.572 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:46.604 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:46.646 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package liboath0:amd64. 2024-09-11T08:36:46.672 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../027-liboath0_2.6.7-3build1_amd64.deb ... 2024-09-11T08:36:46.713 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T08:36:47.024 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libradosstriper1. 2024-09-11T08:36:47.050 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../028-libradosstriper1_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:47.090 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:47.452 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-common. 2024-09-11T08:36:47.478 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../029-ceph-common_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:47.510 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:48.269 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-base. 2024-09-11T08:36:48.295 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:48.345 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:48.907 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-base. 2024-09-11T08:36:48.933 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../030-ceph-base_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:48.940 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T08:36:48.966 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T08:36:48.983 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:49.006 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T08:36:49.284 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T08:36:49.300 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T08:36:49.334 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T08:36:49.553 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.functools. 2024-09-11T08:36:49.579 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../031-python3-jaraco.functools_3.4.0-2_all.deb ... 2024-09-11T08:36:49.595 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T08:36:49.619 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.functools (3.4.0-2) ... 2024-09-11T08:36:49.622 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T08:36:49.654 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T08:36:49.922 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cheroot. 2024-09-11T08:36:49.923 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T08:36:49.948 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../032-python3-cheroot_8.5.2+ds1-1ubuntu3.1_all.deb ... 2024-09-11T08:36:49.950 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T08:36:49.982 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T08:36:49.989 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T08:36:50.268 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T08:36:50.292 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.classes. 2024-09-11T08:36:50.295 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T08:36:50.318 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../033-python3-jaraco.classes_3.2.1-3_all.deb ... 2024-09-11T08:36:50.335 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T08:36:50.358 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.classes (3.2.1-3) ... 2024-09-11T08:36:50.594 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.text. 2024-09-11T08:36:50.621 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../034-python3-jaraco.text_3.6.0-2_all.deb ... 2024-09-11T08:36:50.638 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T08:36:50.661 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.text (3.6.0-2) ... 2024-09-11T08:36:50.664 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T08:36:50.721 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T08:36:50.964 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jaraco.collections. 2024-09-11T08:36:50.991 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../035-python3-jaraco.collections_3.4.0-2_all.deb ... 2024-09-11T08:36:51.007 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-portend. 2024-09-11T08:36:51.022 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T08:36:51.030 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jaraco.collections (3.4.0-2) ... 2024-09-11T08:36:51.057 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T08:36:51.327 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T08:36:51.333 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-tempora. 2024-09-11T08:36:51.353 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T08:36:51.360 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../036-python3-tempora_4.1.2-1_all.deb ... 2024-09-11T08:36:51.393 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T08:36:51.400 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-tempora (4.1.2-1) ... 2024-09-11T08:36:51.694 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-portend. 2024-09-11T08:36:51.696 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T08:36:51.721 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../037-python3-portend_3.0.0-1_all.deb ... 2024-09-11T08:36:51.723 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T08:36:51.752 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-portend (3.0.0-1) ... 2024-09-11T08:36:51.763 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T08:36:52.039 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-zc.lockfile. 2024-09-11T08:36:52.052 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../038-python3-zc.lockfile_2.0-1_all.deb ... 2024-09-11T08:36:52.097 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-zc.lockfile (2.0-1) ... 2024-09-11T08:36:52.142 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T08:36:52.168 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T08:36:52.208 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T08:36:52.342 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cherrypy3. 2024-09-11T08:36:52.368 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../039-python3-cherrypy3_18.6.1-4_all.deb ... 2024-09-11T08:36:52.408 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cherrypy3 (18.6.1-4) ... 2024-09-11T08:36:52.520 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T08:36:52.546 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T08:36:52.586 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T08:36:52.729 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-natsort. 2024-09-11T08:36:52.755 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../040-python3-natsort_8.0.2-1_all.deb ... 2024-09-11T08:36:52.795 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-natsort (8.0.2-1) ... 2024-09-11T08:36:52.864 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T08:36:52.891 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T08:36:52.931 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T08:36:53.123 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-logutils. 2024-09-11T08:36:53.150 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../041-python3-logutils_0.3.3-8_all.deb ... 2024-09-11T08:36:53.189 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-logutils (0.3.3-8) ... 2024-09-11T08:36:53.209 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-mako. 2024-09-11T08:36:53.235 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T08:36:53.275 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T08:36:53.493 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-markupsafe. 2024-09-11T08:36:53.519 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../042-python3-markupsafe_2.0.1-2build1_amd64.deb ... 2024-09-11T08:36:53.559 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-markupsafe (2.0.1-2build1) ... 2024-09-11T08:36:53.596 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T08:36:53.622 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T08:36:53.662 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T08:36:53.965 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T08:36:53.987 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-mako. 2024-09-11T08:36:53.992 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T08:36:54.013 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../043-python3-mako_1.1.3+ds1-2ubuntu0.1_all.deb ... 2024-09-11T08:36:54.045 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T08:36:54.051 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T08:36:54.356 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-simplegeneric. 2024-09-11T08:36:54.376 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-webob. 2024-09-11T08:36:54.383 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../044-python3-simplegeneric_0.8.1-3_all.deb ... 2024-09-11T08:36:54.402 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T08:36:54.423 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2024-09-11T08:36:54.434 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T08:36:54.726 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-singledispatch. 2024-09-11T08:36:54.729 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T08:36:54.752 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../045-python3-singledispatch_3.4.0.3-3_all.deb ... 2024-09-11T08:36:54.756 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T08:36:54.792 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-singledispatch (3.4.0.3-3) ... 2024-09-11T08:36:54.799 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T08:36:55.087 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-webob. 2024-09-11T08:36:55.099 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T08:36:55.113 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../046-python3-webob_1%3a1.8.6-1.1ubuntu0.1_all.deb ... 2024-09-11T08:36:55.125 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T08:36:55.153 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T08:36:55.165 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T08:36:55.469 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-paste. 2024-09-11T08:36:55.495 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T08:36:55.523 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-waitress. 2024-09-11T08:36:55.535 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T08:36:55.549 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../047-python3-waitress_1.4.4-1.1ubuntu1_all.deb ... 2024-09-11T08:36:55.593 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T08:36:55.847 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T08:36:55.873 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T08:36:55.884 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-tempita. 2024-09-11T08:36:55.911 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../048-python3-tempita_0.5.2-6ubuntu1_all.deb ... 2024-09-11T08:36:55.913 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T08:36:55.950 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T08:36:56.186 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-paste. 2024-09-11T08:36:56.202 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../049-python3-paste_3.5.0+dfsg1-1_all.deb ... 2024-09-11T08:36:56.233 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T08:36:56.245 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T08:36:56.259 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T08:36:56.299 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T08:36:56.556 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2024-09-11T08:36:56.572 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../050-python-pastedeploy-tpl_2.1.1-1_all.deb ... 2024-09-11T08:36:56.577 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T08:36:56.593 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T08:36:56.614 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T08:36:56.627 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T08:36:56.875 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pastedeploy. 2024-09-11T08:36:56.880 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T08:36:56.891 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../051-python3-pastedeploy_2.1.1-1_all.deb ... 2024-09-11T08:36:56.907 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T08:36:56.934 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pastedeploy (2.1.1-1) ... 2024-09-11T08:36:56.939 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T08:36:57.277 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T08:36:57.304 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T08:36:57.344 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T08:36:57.353 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-webtest. 2024-09-11T08:36:57.380 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../052-python3-webtest_2.0.35-1_all.deb ... 2024-09-11T08:36:57.420 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-webtest (2.0.35-1) ... 2024-09-11T08:36:57.655 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-idna. 2024-09-11T08:36:57.681 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T08:36:57.714 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pecan. 2024-09-11T08:36:57.721 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T08:36:57.742 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../053-python3-pecan_1.3.3-4ubuntu2_all.deb ... 2024-09-11T08:36:57.781 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T08:36:58.042 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T08:36:58.068 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T08:36:58.108 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T08:36:58.135 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-certifi. 2024-09-11T08:36:58.163 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../054-python3-certifi_2020.6.20-1_all.deb ... 2024-09-11T08:36:58.202 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-certifi (2020.6.20-1) ... 2024-09-11T08:36:58.436 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-requests. 2024-09-11T08:36:58.463 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T08:36:58.503 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T08:36:58.521 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-idna. 2024-09-11T08:36:58.548 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../055-python3-idna_3.3-1ubuntu0.1_all.deb ... 2024-09-11T08:36:58.588 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T08:36:58.798 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T08:36:58.824 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T08:36:58.857 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-urllib3. 2024-09-11T08:36:58.864 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T08:36:58.884 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../056-python3-urllib3_1.26.5-1~exp1ubuntu0.1_all.deb ... 2024-09-11T08:36:58.915 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T08:36:59.152 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T08:36:59.176 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:36:59.219 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:59.252 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-requests. 2024-09-11T08:36:59.279 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../057-python3-requests_2.25.1+dfsg-2ubuntu0.1_all.deb ... 2024-09-11T08:36:59.335 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T08:36:59.616 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T08:36:59.642 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:36:59.647 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-werkzeug. 2024-09-11T08:36:59.673 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../058-python3-werkzeug_2.0.2+dfsg1-1ubuntu0.22.04.2_all.deb ... 2024-09-11T08:36:59.682 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:36:59.713 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T08:37:00.001 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T08:37:00.008 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2024-09-11T08:37:00.028 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T08:37:00.034 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../059-ceph-mgr-modules-core_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:00.068 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T08:37:00.074 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:00.337 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T08:37:00.364 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T08:37:00.404 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T08:37:00.463 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libsqlite3-mod-ceph. 2024-09-11T08:37:00.489 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../060-libsqlite3-mod-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:00.529 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:00.765 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T08:37:00.792 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:00.832 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:00.857 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-bcrypt. 2024-09-11T08:37:00.884 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../061-python3-bcrypt_3.2.0-1build1_amd64.deb ... 2024-09-11T08:37:00.923 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-bcrypt (3.2.0-1build1) ... 2024-09-11T08:37:01.226 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-openssl. 2024-09-11T08:37:01.253 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../062-python3-openssl_21.0.0-1_all.deb ... 2024-09-11T08:37:01.260 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T08:37:01.287 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:01.301 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-openssl (21.0.0-1) ... 2024-09-11T08:37:01.326 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:01.579 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr. 2024-09-11T08:37:01.606 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../063-ceph-mgr_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:01.637 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:01.905 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T08:37:01.931 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:01.963 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:02.032 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mon. 2024-09-11T08:37:02.058 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../064-ceph-mon_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:02.098 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:03.486 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-osd. 2024-09-11T08:37:03.513 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../065-ceph-osd_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:03.552 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:03.850 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph. 2024-09-11T08:37:03.876 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:03.916 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:04.236 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T08:37:04.262 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:04.302 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:04.639 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T08:37:04.665 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:04.705 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:04.788 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph. 2024-09-11T08:37:04.804 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../066-ceph_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:04.846 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:05.116 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package cephadm. 2024-09-11T08:37:05.124 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-fuse. 2024-09-11T08:37:05.143 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:05.150 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../067-ceph-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:05.183 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:05.190 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:05.511 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T08:37:05.537 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T08:37:05.544 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mds. 2024-09-11T08:37:05.571 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../068-ceph-mds_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:05.577 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T08:37:05.610 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:05.905 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T08:37:05.931 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T08:37:05.971 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:06.013 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package cephadm. 2024-09-11T08:37:06.040 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../069-cephadm_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:06.079 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:06.399 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-asyncssh. 2024-09-11T08:37:06.426 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../070-python3-asyncssh_2.5.0-1_all.deb ... 2024-09-11T08:37:06.465 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-asyncssh (2.5.0-1) ... 2024-09-11T08:37:06.777 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-babel-localedata. 2024-09-11T08:37:06.792 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../071-python-babel-localedata_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T08:37:06.826 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:06.927 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-babel. 2024-09-11T08:37:06.954 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T08:37:06.993 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:07.304 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T08:37:07.331 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T08:37:07.388 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T08:37:07.724 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T08:37:07.750 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:07.790 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:07.806 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-babel. 2024-09-11T08:37:07.833 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../072-python3-babel_2.8.0+dfsg.1-7_all.deb ... 2024-09-11T08:37:07.872 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:08.118 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T08:37:08.145 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T08:37:08.185 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T08:37:08.192 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-jinja2. 2024-09-11T08:37:08.219 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../073-python3-jinja2_3.0.3-1ubuntu0.2_all.deb ... 2024-09-11T08:37:08.275 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T08:37:08.463 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-routes. 2024-09-11T08:37:08.489 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T08:37:08.529 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T08:37:08.612 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2024-09-11T08:37:08.639 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../074-ceph-mgr-cephadm_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:08.678 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:08.857 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T08:37:08.884 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:08.915 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:09.006 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-repoze.lru. 2024-09-11T08:37:09.034 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../075-python3-repoze.lru_0.7-2_all.deb ... 2024-09-11T08:37:09.073 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2024-09-11T08:37:09.376 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-routes. 2024-09-11T08:37:09.402 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../076-python3-routes_2.5.1-1ubuntu1_all.deb ... 2024-09-11T08:37:09.442 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T08:37:09.787 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2024-09-11T08:37:09.814 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../077-ceph-mgr-dashboard_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:09.853 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:10.154 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T08:37:10.170 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T08:37:10.212 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T08:37:10.724 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T08:37:10.751 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T08:37:10.790 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T08:37:11.075 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-sklearn-lib:amd64. 2024-09-11T08:37:11.090 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../078-python3-sklearn-lib_0.23.2-5ubuntu6_amd64.deb ... 2024-09-11T08:37:11.133 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T08:37:11.161 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T08:37:11.188 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T08:37:11.227 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T08:37:11.530 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T08:37:11.557 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T08:37:11.596 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T08:37:11.637 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-joblib. 2024-09-11T08:37:11.663 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../079-python3-joblib_0.17.0-4ubuntu1_all.deb ... 2024-09-11T08:37:11.703 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T08:37:12.039 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-threadpoolctl. 2024-09-11T08:37:12.055 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../080-python3-threadpoolctl_3.1.0-1_all.deb ... 2024-09-11T08:37:12.089 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-threadpoolctl (3.1.0-1) ... 2024-09-11T08:37:12.313 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T08:37:12.341 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:12.359 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-sklearn. 2024-09-11T08:37:12.385 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../081-python3-sklearn_0.23.2-5ubuntu6_all.deb ... 2024-09-11T08:37:12.425 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T08:37:12.456 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:13.109 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2024-09-11T08:37:13.136 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../082-ceph-mgr-diskprediction-local_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:13.252 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:13.402 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T08:37:13.429 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T08:37:13.468 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T08:37:13.729 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T08:37:13.760 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T08:37:13.938 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T08:37:14.189 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-cachetools. 2024-09-11T08:37:14.216 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../083-python3-cachetools_5.0.0-1_all.deb ... 2024-09-11T08:37:14.255 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-cachetools (5.0.0-1) ... 2024-09-11T08:37:14.274 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T08:37:14.302 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T08:37:14.341 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T08:37:14.558 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyasn1. 2024-09-11T08:37:14.585 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../084-python3-pyasn1_0.4.8-1_all.deb ... 2024-09-11T08:37:14.624 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyasn1 (0.4.8-1) ... 2024-09-11T08:37:14.669 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T08:37:14.696 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T08:37:14.735 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T08:37:15.039 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T08:37:15.067 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T08:37:15.105 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T08:37:15.178 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyasn1-modules. 2024-09-11T08:37:15.205 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../085-python3-pyasn1-modules_0.2.1-1_all.deb ... 2024-09-11T08:37:15.245 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T08:37:15.425 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T08:37:15.453 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T08:37:15.491 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T08:37:15.556 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-rsa. 2024-09-11T08:37:15.583 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../086-python3-rsa_4.8-1_all.deb ... 2024-09-11T08:37:15.622 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-rsa (4.8-1) ... 2024-09-11T08:37:15.803 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T08:37:15.831 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T08:37:15.870 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T08:37:15.909 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-google-auth. 2024-09-11T08:37:15.936 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../087-python3-google-auth_1.5.1-3_all.deb ... 2024-09-11T08:37:15.967 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-google-auth (1.5.1-3) ... 2024-09-11T08:37:16.189 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T08:37:16.217 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T08:37:16.245 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-requests-oauthlib. 2024-09-11T08:37:16.256 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T08:37:16.272 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../088-python3-requests-oauthlib_1.3.0+ds-0.1_all.deb ... 2024-09-11T08:37:16.312 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T08:37:16.606 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-websocket. 2024-09-11T08:37:16.633 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../089-python3-websocket_1.2.3-1_all.deb ... 2024-09-11T08:37:16.673 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-websocket (1.2.3-1) ... 2024-09-11T08:37:16.886 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T08:37:16.915 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:16.953 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:16.984 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-kubernetes. 2024-09-11T08:37:17.012 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../090-python3-kubernetes_12.0.1-1ubuntu1_all.deb ... 2024-09-11T08:37:17.051 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T08:37:17.222 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T08:37:17.249 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T08:37:17.280 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T08:37:17.634 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T08:37:17.639 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2024-09-11T08:37:17.655 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../091-ceph-mgr-k8sevents_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:17.660 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T08:37:17.689 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:17.700 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T08:37:17.961 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package jq. 2024-09-11T08:37:17.977 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T08:37:17.984 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libonig5:amd64. 2024-09-11T08:37:18.011 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../092-libonig5_6.9.7.1-2build1_amd64.deb ... 2024-09-11T08:37:18.019 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T08:37:18.050 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T08:37:18.272 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package socat. 2024-09-11T08:37:18.299 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T08:37:18.338 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T08:37:18.395 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libjq1:amd64. 2024-09-11T08:37:18.424 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../093-libjq1_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T08:37:18.461 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T08:37:18.617 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T08:37:18.634 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T08:37:18.666 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T08:37:18.722 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package jq. 2024-09-11T08:37:18.738 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../094-jq_1.6-2.1ubuntu3_amd64.deb ... 2024-09-11T08:37:18.772 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking jq (1.6-2.1ubuntu3) ... 2024-09-11T08:37:18.983 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package socat. 2024-09-11T08:37:18.999 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../095-socat_1.7.4.1-3ubuntu4_amd64.deb ... 2024-09-11T08:37:19.012 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-test. 2024-09-11T08:37:19.028 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:19.033 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking socat (1.7.4.1-3ubuntu4) ... 2024-09-11T08:37:19.062 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:19.328 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package xmlstarlet. 2024-09-11T08:37:19.344 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../096-xmlstarlet_1.6.1-2.1_amd64.deb ... 2024-09-11T08:37:19.386 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking xmlstarlet (1.6.1-2.1) ... 2024-09-11T08:37:19.707 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-test. 2024-09-11T08:37:19.722 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../097-ceph-test_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:19.757 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:21.325 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T08:37:21.353 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:21.391 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:21.695 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T08:37:21.711 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:21.753 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:22.039 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T08:37:22.053 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package ceph-volume. 2024-09-11T08:37:22.067 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T08:37:22.081 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../098-ceph-volume_19.1.1-298-gb53b6d3e-1jammy_all.deb ... 2024-09-11T08:37:22.097 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T08:37:22.119 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:22.410 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T08:37:22.438 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T08:37:22.440 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package libcephfs-dev. 2024-09-11T08:37:22.467 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../099-libcephfs-dev_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:22.477 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T08:37:22.506 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:22.834 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua-socket:amd64. 2024-09-11T08:37:22.839 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T08:37:22.861 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../100-lua-socket_3.0~rc1+git+ac3201d-6_amd64.deb ... 2024-09-11T08:37:22.867 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T08:37:22.900 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T08:37:22.905 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T08:37:23.213 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package lua-sec:amd64. 2024-09-11T08:37:23.241 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../101-lua-sec_1.0.2-1_amd64.deb ... 2024-09-11T08:37:23.279 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking lua-sec:amd64 (1.0.2-1) ... 2024-09-11T08:37:23.327 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package pkg-config. 2024-09-11T08:37:23.355 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T08:37:23.394 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T08:37:23.647 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T08:37:23.674 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T08:37:23.680 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package nvme-cli. 2024-09-11T08:37:23.708 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../102-nvme-cli_1.16-3ubuntu0.2_amd64.deb ... 2024-09-11T08:37:23.713 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T08:37:23.800 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T08:37:24.133 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T08:37:24.160 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T08:37:24.163 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package pkg-config. 2024-09-11T08:37:24.191 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../103-pkg-config_0.29.2-1ubuntu3_amd64.deb ... 2024-09-11T08:37:24.199 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T08:37:24.221 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T08:37:24.444 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T08:37:24.457 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python-asyncssh-doc. 2024-09-11T08:37:24.471 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T08:37:24.473 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../104-python-asyncssh-doc_2.5.0-1_all.deb ... 2024-09-11T08:37:24.510 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T08:37:24.516 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python-asyncssh-doc (2.5.0-1) ... 2024-09-11T08:37:24.797 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T08:37:24.825 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T08:37:24.863 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T08:37:24.902 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-iniconfig. 2024-09-11T08:37:24.918 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../105-python3-iniconfig_1.1.1-2_all.deb ... 2024-09-11T08:37:24.952 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-iniconfig (1.1.1-2) ... 2024-09-11T08:37:25.175 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T08:37:25.202 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T08:37:25.241 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T08:37:25.257 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pastescript. 2024-09-11T08:37:25.284 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../106-python3-pastescript_2.0.2-4_all.deb ... 2024-09-11T08:37:25.323 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2024-09-11T08:37:25.586 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-py. 2024-09-11T08:37:25.613 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T08:37:25.626 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pluggy. 2024-09-11T08:37:25.652 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T08:37:25.653 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../107-python3-pluggy_0.13.0-7.1_all.deb ... 2024-09-11T08:37:25.693 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pluggy (0.13.0-7.1) ... 2024-09-11T08:37:25.981 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T08:37:26.004 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-psutil. 2024-09-11T08:37:26.008 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T08:37:26.031 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../108-python3-psutil_5.9.0-1build1_amd64.deb ... 2024-09-11T08:37:26.047 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T08:37:26.070 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-psutil (5.9.0-1build1) ... 2024-09-11T08:37:26.449 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-py. 2024-09-11T08:37:26.476 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../109-python3-py_1.10.0-1_all.deb ... 2024-09-11T08:37:26.501 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T08:37:26.515 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-py (1.10.0-1) ... 2024-09-11T08:37:26.528 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T08:37:26.567 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T08:37:26.812 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-toml. 2024-09-11T08:37:26.826 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pygments. 2024-09-11T08:37:26.827 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T08:37:26.854 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../110-python3-pygments_2.11.2+dfsg-2_all.deb ... 2024-09-11T08:37:26.861 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T08:37:26.892 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T08:37:27.156 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T08:37:27.184 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T08:37:27.222 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T08:37:27.355 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pyinotify. 2024-09-11T08:37:27.383 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../111-python3-pyinotify_0.9.6-1.3_all.deb ... 2024-09-11T08:37:27.421 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pyinotify (0.9.6-1.3) ... 2024-09-11T08:37:27.568 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T08:37:27.584 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T08:37:27.617 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T08:37:27.724 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-toml. 2024-09-11T08:37:27.751 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../112-python3-toml_0.10.2-1_all.deb ... 2024-09-11T08:37:27.807 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-toml (0.10.2-1) ... 2024-09-11T08:37:27.895 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T08:37:27.923 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T08:37:27.953 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T08:37:28.068 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-pytest. 2024-09-11T08:37:28.096 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../113-python3-pytest_6.2.5-1ubuntu2_all.deb ... 2024-09-11T08:37:28.127 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T08:37:28.463 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package python3-simplejson. 2024-09-11T08:37:28.491 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../114-python3-simplejson_3.17.6-1build1_amd64.deb ... 2024-09-11T08:37:28.538 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking python3-simplejson (3.17.6-1build1) ... 2024-09-11T08:37:28.624 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package radosgw. 2024-09-11T08:37:28.652 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:28.690 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:28.815 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package qttranslations5-l10n. 2024-09-11T08:37:28.831 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../115-qttranslations5-l10n_5.15.3-1_all.deb ... 2024-09-11T08:37:28.873 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking qttranslations5-l10n (5.15.3-1) ... 2024-09-11T08:37:29.527 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T08:37:29.554 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:29.577 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package radosgw. 2024-09-11T08:37:29.593 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:29.605 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../116-radosgw_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:29.644 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:29.930 INFO:teuthology.orchestra.run.smithi073.stdout:Selecting previously unselected package rbd-mirror. 2024-09-11T08:37:29.957 INFO:teuthology.orchestra.run.smithi073.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:29.996 INFO:teuthology.orchestra.run.smithi073.stdout:Unpacking rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:30.442 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T08:37:30.472 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package rbd-fuse. 2024-09-11T08:37:30.500 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../117-rbd-fuse_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:30.693 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T08:37:30.763 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:30.819 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T08:37:31.066 INFO:teuthology.orchestra.run.smithi077.stdout:Selecting previously unselected package rbd-mirror. 2024-09-11T08:37:31.085 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T08:37:31.093 INFO:teuthology.orchestra.run.smithi077.stdout:Preparing to unpack .../118-rbd-mirror_19.1.1-298-gb53b6d3e-1jammy_amd64.deb ... 2024-09-11T08:37:31.132 INFO:teuthology.orchestra.run.smithi077.stdout:Unpacking rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:31.435 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T08:37:31.578 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-iniconfig (1.1.1-2) ... 2024-09-11T08:37:31.837 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libdouble-conversion3:amd64 (3.1.7-4) ... 2024-09-11T08:37:31.864 INFO:teuthology.orchestra.run.smithi073.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T08:37:31.937 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:31.955 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up nvme-cli (1.16-3ubuntu0.2) ... 2024-09-11T08:37:32.201 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmefc-boot-connections.service -> /lib/systemd/system/nvmefc-boot-connections.service. 2024-09-11T08:37:32.429 INFO:teuthology.orchestra.run.smithi073.stdout:Adding system user cephadm....done 2024-09-11T08:37:32.524 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/default.target.wants/nvmf-autoconnect.service -> /lib/systemd/system/nvmf-autoconnect.service. 2024-09-11T08:37:32.593 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T08:37:32.886 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T08:37:32.941 INFO:teuthology.orchestra.run.smithi077.stdout:nvmf-connect.target is a disabled or a static unit, not starting it. 2024-09-11T08:37:33.007 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:33.146 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T08:37:33.247 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T08:37:33.489 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T08:37:33.531 INFO:teuthology.orchestra.run.smithi077.stdout:Adding system user cephadm....done 2024-09-11T08:37:33.688 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-waitress (1.4.4-1.1ubuntu1) ... 2024-09-11T08:37:33.740 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T08:37:33.842 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T08:37:33.956 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.classes (3.2.1-3) ... 2024-09-11T08:37:34.127 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T08:37:34.190 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-asyncssh-doc (2.5.0-1) ... 2024-09-11T08:37:34.316 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.functools (3.4.0-2) ... 2024-09-11T08:37:34.495 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T08:37:34.567 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-repoze.lru (0.7-2) ... 2024-09-11T08:37:34.771 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T08:37:34.826 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liboath0:amd64 (2.6.7-3build1) ... 2024-09-11T08:37:34.952 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-py (1.10.0-1) ... 2024-09-11T08:37:35.031 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T08:37:35.254 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-joblib (0.17.0-4ubuntu1) ... 2024-09-11T08:37:35.265 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T08:37:35.517 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T08:37:35.598 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cachetools (5.0.0-1) ... 2024-09-11T08:37:35.759 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T08:37:35.832 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-openssl (21.0.0-1) ... 2024-09-11T08:37:36.019 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:36.295 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T08:37:36.300 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up unzip (6.0-26ubuntu3.2) ... 2024-09-11T08:37:36.421 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T08:37:36.543 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-bcrypt (3.2.0-1build1) ... 2024-09-11T08:37:36.547 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T08:37:36.673 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T08:37:36.803 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyinotify (0.9.6-1.3) ... 2024-09-11T08:37:36.923 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T08:37:37.062 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-threadpoolctl (3.1.0-1) ... 2024-09-11T08:37:37.307 INFO:teuthology.orchestra.run.smithi073.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T08:37:37.313 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-ceph-argparse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:37.355 INFO:teuthology.orchestra.run.smithi073.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T08:37:37.419 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T08:37:37.545 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T08:37:37.547 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-sklearn-lib:amd64 (0.23.2-5ubuntu6) ... 2024-09-11T08:37:37.649 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-6) ... 2024-09-11T08:37:37.799 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libreadline-dev:amd64 (8.1.2-1) ... 2024-09-11T08:37:37.904 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:37.925 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-markupsafe (2.0.1-2build1) ... 2024-09-11T08:37:38.005 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T08:37:38.185 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua5.1 (5.1.5-8.1build4) ... 2024-09-11T08:37:38.256 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T08:37:38.507 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T08:37:38.567 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/lua5.1 to provide /usr/bin/lua (lua-interpreter) in auto mode 2024-09-11T08:37:38.598 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/luac5.1 to provide /usr/bin/luac (lua-compiler) in auto mode 2024-09-11T08:37:38.662 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libpcre2-16-0:amd64 (10.39-3ubuntu0.1) ... 2024-09-11T08:37:38.772 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-psutil (5.9.0-1build1) ... 2024-09-11T08:37:38.809 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T08:37:38.918 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T08:37:39.115 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-babel-localedata (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:39.233 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-natsort (8.0.2-1) ... 2024-09-11T08:37:39.500 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-routes (2.5.1-1ubuntu1) ... 2024-09-11T08:37:39.570 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T08:37:39.768 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-simplejson (3.17.6-1build1) ... 2024-09-11T08:37:39.829 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T08:37:39.930 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T08:37:40.040 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T08:37:40.086 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up zip (3.0-12build2) ... 2024-09-11T08:37:40.212 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pygments (2.11.2+dfsg-2) ... 2024-09-11T08:37:40.332 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T08:37:40.717 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T08:37:40.897 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-tempita (0.5.2-6ubuntu1) ... 2024-09-11T08:37:40.959 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T08:37:41.131 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python-pastedeploy-tpl (2.1.1-1) ... 2024-09-11T08:37:41.249 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up qttranslations5-l10n (5.15.3-1) ... 2024-09-11T08:37:41.320 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T08:37:41.375 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-wcwidth (0.2.5+dfsg1-1) ... 2024-09-11T08:37:41.612 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T08:37:41.684 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-asyncssh (2.5.0-1) ... 2024-09-11T08:37:41.964 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T08:37:42.069 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-certifi (2020.6.20-1) ... 2024-09-11T08:37:42.223 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T08:37:42.320 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-paste (3.5.0+dfsg1-1) ... 2024-09-11T08:37:42.324 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:42.609 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T08:37:42.721 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cheroot (8.5.2+ds1-1ubuntu3.1) ... 2024-09-11T08:37:42.912 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T08:37:42.998 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-werkzeug (2.0.2+dfsg1-1ubuntu0.22.04.2) ... 2024-09-11T08:37:43.324 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.text (3.6.0-2) ... 2024-09-11T08:37:43.575 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up socat (1.7.4.1-3ubuntu4) ... 2024-09-11T08:37:43.701 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:44.002 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-idna (3.3-1ubuntu0.1) ... 2024-09-11T08:37:44.009 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T08:37:44.219 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:37:44.320 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-sklearn (0.23.2-5ubuntu6) ... 2024-09-11T08:37:44.345 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T08:37:44.840 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T08:37:44.964 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T08:37:45.090 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T08:37:45.383 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T08:37:45.484 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up pkg-config (0.29.2-1ubuntu3) ... 2024-09-11T08:37:45.618 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T08:37:45.735 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libqt5core5a:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:37:45.851 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:37:45.869 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-toml (0.10.2-1) ... 2024-09-11T08:37:45.977 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T08:37:46.145 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librdkafka1:amd64 (1.8.0-1build1) ... 2024-09-11T08:37:46.246 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up xmlstarlet (1.6.1-2.1) ... 2024-09-11T08:37:46.296 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T08:37:46.372 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-urllib3 (1.26.5-1~exp1ubuntu0.1) ... 2024-09-11T08:37:46.555 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T08:37:46.665 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pluggy (0.13.0-7.1) ... 2024-09-11T08:37:46.831 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T08:37:46.925 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-zc.lockfile (2.0-1) ... 2024-09-11T08:37:47.084 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T08:37:47.176 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libqt5dbus5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:37:47.302 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyasn1 (0.4.8-1) ... 2024-09-11T08:37:47.324 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T08:37:47.576 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T08:37:47.603 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-singledispatch (3.4.0.3-3) ... 2024-09-11T08:37:47.702 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T08:37:47.854 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-logutils (0.3.3-8) ... 2024-09-11T08:37:47.986 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T08:37:48.105 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-tempora (4.1.2-1) ... 2024-09-11T08:37:48.121 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T08:37:48.340 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2024-09-11T08:37:48.413 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T08:37:48.574 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-prettytable (2.5.0-2) ... 2024-09-11T08:37:48.723 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T08:37:48.841 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liblttng-ust1:amd64 (2.13.1-1ubuntu1) ... 2024-09-11T08:37:48.967 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-websocket (1.2.3-1) ... 2024-09-11T08:37:48.974 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T08:37:49.075 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T08:37:49.184 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:49.252 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libonig5:amd64 (6.9.7.1-2build1) ... 2024-09-11T08:37:49.293 INFO:teuthology.orchestra.run.smithi073.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T08:37:49.378 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-mako (1.1.3+ds1-2ubuntu0.1) ... 2024-09-11T08:37:49.528 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T08:37:49.662 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T08:37:49.663 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-webob (1:1.8.6-1.1ubuntu0.1) ... 2024-09-11T08:37:49.972 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jaraco.collections (3.4.0-2) ... 2024-09-11T08:37:50.055 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T08:37:50.206 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up liblua5.3-dev:amd64 (5.3.6-1build1) ... 2024-09-11T08:37:50.307 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua-sec:amd64 (1.0.2-1) ... 2024-09-11T08:37:50.323 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T08:37:50.424 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T08:37:50.433 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-babel (2.8.0+dfsg.1-7) ... 2024-09-11T08:37:50.522 INFO:teuthology.orchestra.run.smithi077.stdout:update-alternatives: using /usr/bin/pybabel-python3 to provide /usr/bin/pybabel (pybabel) in auto mode 2024-09-11T08:37:50.658 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:37:50.759 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T08:37:50.760 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libjq1:amd64 (1.6-2.1ubuntu3) ... 2024-09-11T08:37:50.886 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pytest (6.2.5-1ubuntu2) ... 2024-09-11T08:37:51.085 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T08:37:51.262 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pastedeploy (2.1.1-1) ... 2024-09-11T08:37:51.386 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T08:37:51.663 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T08:37:51.711 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up lua-any (27ubuntu1) ... 2024-09-11T08:37:51.789 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T08:37:51.889 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-portend (3.0.0-1) ... 2024-09-11T08:37:52.057 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T08:37:52.148 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libqt5network5:amd64 (5.15.3+dfsg-2ubuntu0.2) ... 2024-09-11T08:37:52.274 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-jinja2 (3.0.3-1ubuntu0.2) ... 2024-09-11T08:37:52.316 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T08:37:52.600 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pyasn1-modules (0.2.1-1) ... 2024-09-11T08:37:52.636 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T08:37:52.910 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-requests (2.25.1+dfsg-2ubuntu0.1) ... 2024-09-11T08:37:53.021 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T08:37:53.178 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up jq (1.6-2.1ubuntu3) ... 2024-09-11T08:37:53.278 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rsa (4.8-1) ... 2024-09-11T08:37:53.338 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T08:37:53.546 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-webtest (2.0.35-1) ... 2024-09-11T08:37:53.691 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T08:37:53.792 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:53.814 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-requests-oauthlib (1.3.0+ds-0.1) ... 2024-09-11T08:37:53.893 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:54.036 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T08:37:54.150 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cherrypy3 (18.6.1-4) ... 2024-09-11T08:37:54.372 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:54.498 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:54.510 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pastescript (2.0.2-4) ... 2024-09-11T08:37:54.624 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T08:37:54.795 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-pecan (1.3.3-4ubuntu2) ... 2024-09-11T08:37:54.892 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:54.993 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:55.111 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:55.139 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libthrift-0.16.0:amd64 (0.16.0-2) ... 2024-09-11T08:37:55.249 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librados2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:55.333 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T08:37:55.333 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T08:37:55.375 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libsqlite3-mod-ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:55.501 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up luarocks (3.8.0+dfsg1-1) ... 2024-09-11T08:37:55.829 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:55.837 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libcephfs2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:55.955 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:55.963 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libradosstriper1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:56.072 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-google-auth (1.5.1-3) ... 2024-09-11T08:37:56.081 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:56.207 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T08:37:56.332 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librbd1 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:56.433 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-modules-core (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:56.534 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:56.749 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T08:37:56.749 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2024-09-11T08:37:57.227 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up libcephfs-dev (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:57.353 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rados (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:57.444 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:57.479 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up librgw2 (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:57.579 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:57.605 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-kubernetes (12.0.1-1ubuntu1) ... 2024-09-11T08:37:57.705 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:57.830 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:57.940 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:58.416 INFO:teuthology.orchestra.run.smithi073.stdout:Adding group ceph....done 2024-09-11T08:37:58.784 INFO:teuthology.orchestra.run.smithi073.stdout:Adding system user ceph....done 2024-09-11T08:37:58.867 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rbd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:58.944 INFO:teuthology.orchestra.run.smithi073.stdout:Setting system user ceph properties....done 2024-09-11T08:37:58.961 INFO:teuthology.orchestra.run.smithi073.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T08:37:58.976 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up rbd-fuse (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:59.102 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-rgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:59.108 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T08:37:59.228 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up python3-cephfs (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:59.354 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-common (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:37:59.465 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T08:37:59.880 INFO:teuthology.orchestra.run.smithi077.stdout:Adding group ceph....done 2024-09-11T08:37:59.949 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:00.067 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:00.248 INFO:teuthology.orchestra.run.smithi077.stdout:Adding system user ceph....done 2024-09-11T08:38:00.358 INFO:teuthology.orchestra.run.smithi077.stdout:Setting system user ceph properties....done 2024-09-11T08:38:00.373 INFO:teuthology.orchestra.run.smithi077.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2024-09-11T08:38:00.517 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2024-09-11T08:38:00.523 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:38:00.523 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:38:00.881 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2024-09-11T08:38:00.985 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:01.330 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T08:38:01.354 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-test (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:01.489 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up radosgw (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:01.821 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:01.974 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:38:01.974 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2024-09-11T08:38:02.040 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T08:38:02.041 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T08:38:02.432 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-base (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:02.497 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:02.707 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:38:02.707 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:38:02.787 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2024-09-11T08:38:03.165 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:03.250 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mds (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:03.363 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T08:38:03.363 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T08:38:03.474 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T08:38:03.474 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2024-09-11T08:38:03.866 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:03.936 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up rbd-mirror (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:04.155 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:38:04.155 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target -> /lib/systemd/system/ceph-rbd-mirror.target. 2024-09-11T08:38:04.162 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T08:38:04.162 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T08:38:04.626 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:04.628 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:04.744 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:04.844 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T08:38:04.845 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2024-09-11T08:38:04.895 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:05.108 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T08:38:05.108 INFO:teuthology.orchestra.run.smithi073.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T08:38:05.296 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-osd (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:05.563 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T08:38:05.563 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2024-09-11T08:38:05.605 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:05.891 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:05.998 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-k8sevents (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:06.043 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:06.124 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-diskprediction-local (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:06.193 INFO:teuthology.orchestra.run.smithi073.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:06.275 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mon (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:06.488 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T08:38:06.488 INFO:teuthology.orchestra.run.smithi077.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2024-09-11T08:38:06.548 INFO:teuthology.orchestra.run.smithi073.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T08:38:06.700 INFO:teuthology.orchestra.run.smithi073.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T08:38:06.968 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-cephadm (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:06.978 INFO:teuthology.orchestra.run.smithi073.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T08:38:07.163 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:07.570 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-mgr-dashboard (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:07.723 INFO:teuthology.orchestra.run.smithi077.stdout:Setting up ceph-volume (19.1.1-298-gb53b6d3e-1jammy) ... 2024-09-11T08:38:08.076 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for mailcap (3.70+nmu1ubuntu1) ... 2024-09-11T08:38:08.150 INFO:teuthology.orchestra.run.smithi073.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T08:38:08.185 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for libc-bin (2.35-0ubuntu3.1) ... 2024-09-11T08:38:08.337 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for man-db (2.10.2-1) ... 2024-09-11T08:38:08.763 INFO:teuthology.orchestra.run.smithi073.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T08:38:08.772 DEBUG:teuthology.parallel:result is None 2024-09-11T08:38:09.471 INFO:teuthology.orchestra.run.smithi077.stdout:Processing triggers for install-info (6.8-4build1) ... 2024-09-11T08:38:10.139 INFO:teuthology.orchestra.run.smithi077.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2024-09-11T08:38:10.144 DEBUG:teuthology.parallel:result is None 2024-09-11T08:38:10.145 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T08:38:10.295 DEBUG:teuthology.orchestra.run.smithi073:> dpkg-query -W -f '${Version}' ceph 2024-09-11T08:38:10.315 INFO:teuthology.orchestra.run.smithi073.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:38:10.315 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:38:10.315 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T08:38:10.317 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F22.04%2Fx86_64&sha1=b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc 2024-09-11T08:38:10.461 DEBUG:teuthology.orchestra.run.smithi077:> dpkg-query -W -f '${Version}' ceph 2024-09-11T08:38:10.479 INFO:teuthology.orchestra.run.smithi077.stdout:19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:38:10.480 INFO:teuthology.packaging:The installed version of ceph is 19.1.1-298-gb53b6d3e-1jammy 2024-09-11T08:38:10.480 INFO:teuthology.task.install:The correct ceph version 19.1.1-298-gb53b6d3e-1jammy is installed. 2024-09-11T08:38:10.481 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-09-11T08:38:10.482 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:10.482 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T08:38:10.496 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:10.496 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-09-11T08:38:10.538 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-09-11T08:38:10.538 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:10.538 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T08:38:10.552 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T08:38:10.605 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:10.606 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/daemon-helper 2024-09-11T08:38:10.619 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-09-11T08:38:10.675 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-09-11T08:38:10.675 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:10.675 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T08:38:10.688 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T08:38:10.741 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:10.742 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/adjust-ulimits 2024-09-11T08:38:10.754 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-09-11T08:38:10.806 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-09-11T08:38:10.807 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:10.807 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T08:38:10.820 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T08:38:10.873 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:10.874 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/usr/bin/stdin-killer 2024-09-11T08:38:10.887 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-09-11T08:38:10.938 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T08:38:11.019 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2024-09-11T08:38:11.019 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 777 /var/log/ceph 2024-09-11T08:38:11.022 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 777 /var/log/ceph 2024-09-11T08:38:11.035 INFO:tasks.ceph:Disabling ceph logrotate... 2024-09-11T08:38:11.035 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T08:38:11.076 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/logrotate.d/ceph 2024-09-11T08:38:11.088 INFO:tasks.ceph:Creating extra log directories... 2024-09-11T08:38:11.088 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T08:38:11.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2024-09-11T08:38:11.157 INFO:tasks.ceph:Creating ceph cluster cluster1... 2024-09-11T08:38:11.157 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster1', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T08:38:11.157 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900659', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi073.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuh0QeHx3fryzEpQpK5xbTHvxwgQg32edZPQTXOSJGseSUdnXvrij6JDuffVEHtSPdwy3PgxAkZctvxRi6AXoo=', 'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZ2WuzHB7zjUUpe0P9YBQNHv5XrPIF9koGEMtfiHnIw/SlLQrufkVBxayqxTGeezl22EHKhHzsda1D5Jb1zis4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster2'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T08:38:11.158 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-11T08:38:11.188 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/cluster1.data 2024-09-11T08:38:11.193 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T08:38:11.236 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T08:38:11.249 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:11.249 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T08:38:11.295 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T08:38:11.295 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_1 2024-09-11T08:38:11.347 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T08:38:11.347 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:11.347 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 875 Links: 1 2024-09-11T08:38:11.347 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:11.347 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-09-11 08:32:50.307913964 +0000 2024-09-11T08:38:11.347 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-09-11 08:32:50.067916093 +0000 2024-09-11T08:38:11.348 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-09-11 08:32:50.067916093 +0000 2024-09-11T08:38:11.348 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2024-09-11T08:38:11.348 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T08:38:11.401 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-09-11T08:38:11.402 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-09-11T08:38:11.402 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000443497 s, 1.2 MB/s 2024-09-11T08:38:11.403 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T08:38:11.452 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_2 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 893 Links: 1 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-09-11 08:32:50.783909737 +0000 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-09-11 08:32:50.527912011 +0000 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-09-11 08:32:50.527912011 +0000 2024-09-11T08:38:11.499 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2024-09-11T08:38:11.500 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T08:38:11.553 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-09-11T08:38:11.553 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-09-11T08:38:11.553 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000374824 s, 1.4 MB/s 2024-09-11T08:38:11.555 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T08:38:11.604 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_3 2024-09-11T08:38:11.651 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T08:38:11.651 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:11.651 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 912 Links: 1 2024-09-11T08:38:11.651 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:11.651 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-09-11 08:32:51.243905654 +0000 2024-09-11T08:38:11.651 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-09-11 08:32:51.007907750 +0000 2024-09-11T08:38:11.651 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-09-11 08:32:51.007907750 +0000 2024-09-11T08:38:11.652 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2024-09-11T08:38:11.652 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T08:38:11.705 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-09-11T08:38:11.705 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-09-11T08:38:11.706 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000435603 s, 1.2 MB/s 2024-09-11T08:38:11.707 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T08:38:11.756 DEBUG:teuthology.orchestra.run.smithi073:> stat /dev/vg_nvme/lv_4 2024-09-11T08:38:11.807 INFO:teuthology.orchestra.run.smithi073.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T08:38:11.807 INFO:teuthology.orchestra.run.smithi073.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:11.807 INFO:teuthology.orchestra.run.smithi073.stdout:Device: 5h/5d Inode: 930 Links: 1 2024-09-11T08:38:11.807 INFO:teuthology.orchestra.run.smithi073.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:11.807 INFO:teuthology.orchestra.run.smithi073.stdout:Access: 2024-09-11 08:32:51.699901606 +0000 2024-09-11T08:38:11.807 INFO:teuthology.orchestra.run.smithi073.stdout:Modify: 2024-09-11 08:32:51.459903736 +0000 2024-09-11T08:38:11.807 INFO:teuthology.orchestra.run.smithi073.stdout:Change: 2024-09-11 08:32:51.459903736 +0000 2024-09-11T08:38:11.808 INFO:teuthology.orchestra.run.smithi073.stdout: Birth: - 2024-09-11T08:38:11.808 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T08:38:11.861 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records in 2024-09-11T08:38:11.861 INFO:teuthology.orchestra.run.smithi073.stderr:1+0 records out 2024-09-11T08:38:11.861 INFO:teuthology.orchestra.run.smithi073.stderr:512 bytes copied, 0.000411091 s, 1.2 MB/s 2024-09-11T08:38:11.863 DEBUG:teuthology.orchestra.run.smithi073:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T08:38:11.912 INFO:tasks.ceph:osd dev map: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:11.912 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T08:38:11.912 INFO:tasks.ceph:Generating config... 2024-09-11T08:38:11.913 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-11T08:38:11.913 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client] log to stderr = False 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:11.914 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:11.914 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T08:38:11.915 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T08:38:11.915 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T08:38:11.915 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T08:38:11.915 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T08:38:11.915 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T08:38:11.915 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T08:38:11.915 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T08:38:11.915 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-11T08:38:11.915 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-11T08:38:11.915 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T08:38:11.915 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T08:38:11.915 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T08:38:11.915 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-11T08:38:11.916 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T08:38:11.916 INFO:tasks.ceph:Setting up cluster1.mon.a... 2024-09-11T08:38:11.916 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster1.keyring 2024-09-11T08:38:11.984 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/cluster1.keyring 2024-09-11T08:38:11.997 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster1.keyring 2024-09-11T08:38:12.071 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T08:38:12.125 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster1.mon.a', '172.21.15.73')] 2024-09-11T08:38:12.125 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.73', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-11T08:38:12.126 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:12.126 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T08:38:12.171 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.73 --print /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: generated fsid 9661569a-ff40-4cdb-a5be-7b937214efd1 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:setting min_mon_release = quincy 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:epoch 0 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:fsid 9661569a-ff40-4cdb-a5be-7b937214efd1 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:last_changed 2024-09-11T08:38:12.252100+0000 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:created 2024-09-11T08:38:12.252100+0000 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:min_mon_release 17 (quincy) 2024-09-11T08:38:12.253 INFO:teuthology.orchestra.run.smithi073.stdout:election_strategy: 1 2024-09-11T08:38:12.254 INFO:teuthology.orchestra.run.smithi073.stdout:0: [v2:172.21.15.73:3300/0,v1:172.21.15.73:6789/0] mon.a 2024-09-11T08:38:12.254 INFO:teuthology.orchestra.run.smithi073.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster1.monmap (1 monitors) 2024-09-11T08:38:12.254 DEBUG:teuthology.orchestra.run.smithi073:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T08:38:12.299 INFO:tasks.ceph:Writing /etc/ceph/cluster1.conf for FSID 9661569a-ff40-4cdb-a5be-7b937214efd1... 2024-09-11T08:38:12.300 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-11T08:38:12.344 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster1.conf && sudo chmod 0644 /etc/ceph/cluster1.conf > /dev/null 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: chdir = "" 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: auth supported = cephx 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: filestore xattr use omap = true 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: mon clock drift allowed = 1.000 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.372 INFO:teuthology.orchestra.run.smithi073.stdout: osd crush chooseleaf type = 0 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: auth debug = true 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on old message = true 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on bug = true 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: mon pg warn max object skew = 0 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default size = 2 2024-09-11T08:38:12.373 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow primary affinity = true 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow pg remap = true 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on legacy crush tunables = false 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on crush straw calc version zero = false 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on no sortbitwise = false 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on osd down out interval zero = false 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on too few osds = false 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: mon_allow_pool_size_one = true 2024-09-11T08:38:12.374 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: osd default data pool replay window = 5 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: mon allow pool delete = true 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: mon cluster log file level = debug 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: debug asserts on shutdown = true 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: mon health detail to clog = false 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: mon host = 172.21.15.73 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: mon client directed command retry = 5 2024-09-11T08:38:12.375 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on pool no app = False 2024-09-11T08:38:12.376 INFO:teuthology.orchestra.run.smithi073.stdout: ms inject socket failures = 5000 2024-09-11T08:38:12.376 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = 9661569a-ff40-4cdb-a5be-7b937214efd1 2024-09-11T08:38:12.376 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.376 INFO:teuthology.orchestra.run.smithi073.stdout:[osd] 2024-09-11T08:38:12.376 INFO:teuthology.orchestra.run.smithi073.stdout: osd journal size = 100 2024-09-11T08:38:12.376 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub load threshold = 5.0 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub max interval = 600 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd mclock profile = high_recovery_ops 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd recover clone overlap = true 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd recovery max chunk = 1048576 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug shutdown = true 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug op order = true 2024-09-11T08:38:12.377 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug verify stray on activate = true 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug trim objects = true 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: osd open classes on start = true 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug pg log writeout = true 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: osd deep scrub update digest min age = 30 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: osd map max advance = 10 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.378 INFO:teuthology.orchestra.run.smithi073.stdout: journal zero on create = true 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: filestore ondisk finisher threads = 3 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: filestore apply finisher threads = 3 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: bdev debug aio = true 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug misdirected ops = true 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: bdev async discard = True 2024-09-11T08:38:12.379 INFO:teuthology.orchestra.run.smithi073.stdout: bdev enable discard = True 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore block size = 96636764160 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore fsck on mount = True 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluefs = 1/20 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluestore = 1/20 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: debug osd = 20 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: debug rocksdb = 4/10 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd full ratio = 0.9 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd nearfull ratio = 0.8 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: osd failsafe full ratio = 0.95 2024-09-11T08:38:12.380 INFO:teuthology.orchestra.run.smithi073.stdout: osd memory target = 1610612736 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: osd objectstore = bluestore 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr] 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: debug mgr = 20 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: mgr/telemetry/nag = false 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout:[mon] 2024-09-11T08:38:12.381 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: debug paxos = 20 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: mon data avail warn = 5 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: mon mgr mkfs grace = 240 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd reporter subtree level = osd 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd prime pg temp = true 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T08:38:12.382 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: # 1m isn't quite enough 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: mon_down_mkfs_grace = 2m 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_filestore_osds = false 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout:[client] 2024-09-11T08:38:12.383 INFO:teuthology.orchestra.run.smithi073.stdout: rgw cache enabled = true 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable ops log = true 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable usage log = true 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: debug rbd = 20 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: debug rbd_mirror = 15 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: log to stderr = False 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: rbd default features = 125 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: rbd mirror image policy type = simple 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.a] 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.0] 2024-09-11T08:38:12.384 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.1] 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.2] 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.3] 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.385 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.387 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-09-11T08:38:12.388 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-09-11T08:38:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-09-11T08:38:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-09-11T08:38:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-09-11T08:38:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-09-11T08:38:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-09-11T08:38:12.389 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.390 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.73 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: mon client directed command retry = 5 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on pool no app = False 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject socket failures = 5000 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 9661569a-ff40-4cdb-a5be-7b937214efd1 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.391 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.392 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-09-11T08:38:12.393 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-09-11T08:38:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-09-11T08:38:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-09-11T08:38:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-09-11T08:38:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-09-11T08:38:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-09-11T08:38:12.394 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: osd memory target = 1610612736 2024-09-11T08:38:12.395 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-09-11T08:38:12.396 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T08:38:12.397 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-09-11T08:38:12.398 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T08:38:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T08:38:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd = 20 2024-09-11T08:38:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd_mirror = 15 2024-09-11T08:38:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: log to stderr = False 2024-09-11T08:38:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: rbd default features = 125 2024-09-11T08:38:12.399 INFO:teuthology.orchestra.run.smithi077.stdout: rbd mirror image policy type = simple 2024-09-11T08:38:12.400 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-09-11T08:38:12.400 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2024-09-11T08:38:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.400 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2024-09-11T08:38:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.400 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.401 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2024-09-11T08:38:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.401 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2024-09-11T08:38:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:12.401 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:12.401 INFO:tasks.ceph:Creating admin key on cluster1.mon.a... 2024-09-11T08:38:12.401 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster1.keyring 2024-09-11T08:38:12.478 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T08:38:12.478 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:12.478 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/cluster1.keyring of=/dev/stdout 2024-09-11T08:38:12.484 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:12.484 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/home/ubuntu/cephtest/cluster1.monmap of=/dev/stdout 2024-09-11T08:38:12.530 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T08:38:12.531 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:12.531 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-11T08:38:12.531 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T08:38:12.594 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:12.594 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:12.643 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T08:38:12.643 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:12.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/cluster1.keyring 2024-09-11T08:38:12.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/cluster1.keyring 2024-09-11T08:38:12.668 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:12.668 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:12.716 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T08:38:12.717 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T08:38:12.717 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mgr/cluster1-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-11T08:38:12.760 INFO:teuthology.orchestra.run.smithi073.stdout:creating /var/lib/ceph/mgr/cluster1-x/keyring 2024-09-11T08:38:12.763 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T08:38:12.763 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T08:38:12.764 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster1.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.0.keyring 2024-09-11T08:38:12.842 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/cluster1.client.0.keyring 2024-09-11T08:38:12.852 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster1.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.keyring 2024-09-11T08:38:12.887 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.keyring 2024-09-11T08:38:12.934 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster1.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-11T08:38:13.011 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.0.keyring 2024-09-11T08:38:13.022 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster1.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-11T08:38:13.095 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.1.keyring 2024-09-11T08:38:13.105 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster1.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-11T08:38:13.177 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.2.keyring 2024-09-11T08:38:13.186 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster1.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-11T08:38:13.255 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.3.keyring 2024-09-11T08:38:13.264 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster1.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-11T08:38:13.338 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.4.keyring 2024-09-11T08:38:13.349 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster1.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-11T08:38:13.423 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.5.keyring 2024-09-11T08:38:13.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster1.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-11T08:38:13.511 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster1.client.mirror.6.keyring 2024-09-11T08:38:13.521 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T08:38:13.521 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T08:38:13.521 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/cluster1-0 2024-09-11T08:38:13.535 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:13.535 INFO:tasks.ceph:role: cluster1.osd.0 2024-09-11T08:38:13.536 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T08:38:13.536 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T08:38:13.597 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T08:38:13.598 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T08:38:14.069 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2024-09-11T08:38:14.070 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2024-09-11T08:38:14.071 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster1-0 2024-09-11T08:38:14.160 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-0 2024-09-11T08:38:14.169 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T08:38:14.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:38:14.170 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/cluster1-1 2024-09-11T08:38:14.225 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:14.225 INFO:tasks.ceph:role: cluster1.osd.1 2024-09-11T08:38:14.225 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T08:38:14.225 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T08:38:14.285 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T08:38:14.285 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T08:38:14.285 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T08:38:14.286 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T08:38:14.286 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T08:38:14.286 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2024-09-11T08:38:14.286 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T08:38:14.286 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T08:38:14.286 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T08:38:14.286 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T08:38:14.747 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2024-09-11T08:38:14.749 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2024-09-11T08:38:14.749 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster1-1 2024-09-11T08:38:14.878 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-1 2024-09-11T08:38:14.889 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T08:38:14.889 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:38:14.890 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/osd/cluster1-2 2024-09-11T08:38:14.946 INFO:tasks.ceph:roles_to_devs: {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:14.946 INFO:tasks.ceph:role: cluster1.osd.2 2024-09-11T08:38:14.946 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T08:38:14.946 DEBUG:teuthology.orchestra.run.smithi073:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout: = sunit=0 swidth=0 blks 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T08:38:15.014 INFO:teuthology.orchestra.run.smithi073.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T08:38:15.015 INFO:teuthology.orchestra.run.smithi073.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T08:38:15.513 INFO:teuthology.orchestra.run.smithi073.stdout:Discarding blocks...Done. 2024-09-11T08:38:15.516 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi073.front.sepia.ceph.com -o noatime 2024-09-11T08:38:15.516 DEBUG:teuthology.orchestra.run.smithi073:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster1-2 2024-09-11T08:38:15.674 DEBUG:teuthology.orchestra.run.smithi073:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster1-2 2024-09-11T08:38:15.685 INFO:teuthology.orchestra.run.smithi073.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T08:38:15.685 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:38:15.686 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:15.804 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:15.800+0000 7f255f1137c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-0/keyring: can't open /var/lib/ceph/osd/cluster1-0/keyring: (2) No such file or directory 2024-09-11T08:38:15.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:15.800+0000 7f255f1137c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-0/keyring 2024-09-11T08:38:15.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:15.800+0000 7f255f1137c0 -1 bdev(0x564162dc3800 /var/lib/ceph/osd/cluster1-0/block) open stat got: (1) Operation not permitted 2024-09-11T08:38:15.805 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:15.800+0000 7f255f1137c0 -1 bluestore(/var/lib/ceph/osd/cluster1-0) _read_fsid unparsable uuid 2024-09-11T08:38:19.427 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-0 2024-09-11T08:38:19.441 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:19.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:19.520+0000 7ff0dcf207c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-1/keyring: can't open /var/lib/ceph/osd/cluster1-1/keyring: (2) No such file or directory 2024-09-11T08:38:19.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:19.520+0000 7ff0dcf207c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-1/keyring 2024-09-11T08:38:19.525 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:19.520+0000 7ff0dcf207c0 -1 bdev(0x55db22c7d800 /var/lib/ceph/osd/cluster1-1/block) open stat got: (1) Operation not permitted 2024-09-11T08:38:19.526 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:19.520+0000 7ff0dcf207c0 -1 bluestore(/var/lib/ceph/osd/cluster1-1) _read_fsid unparsable uuid 2024-09-11T08:38:23.154 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-1 2024-09-11T08:38:23.211 DEBUG:teuthology.orchestra.run.smithi073:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster1 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:23.295 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:23.292+0000 7f6aeb08b7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster1-2/keyring: can't open /var/lib/ceph/osd/cluster1-2/keyring: (2) No such file or directory 2024-09-11T08:38:23.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:23.292+0000 7f6aeb08b7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster1-2/keyring 2024-09-11T08:38:23.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:23.292+0000 7f6aeb08b7c0 -1 bdev(0x558fc651b800 /var/lib/ceph/osd/cluster1-2/block) open stat got: (1) Operation not permitted 2024-09-11T08:38:23.296 INFO:teuthology.orchestra.run.smithi073.stderr:2024-09-11T08:38:23.292+0000 7f6aeb08b7c0 -1 bluestore(/var/lib/ceph/osd/cluster1-2) _read_fsid unparsable uuid 2024-09-11T08:38:26.662 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster1-2 2024-09-11T08:38:26.719 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T08:38:26.719 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:26.719 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/mgr/cluster1-x/keyring of=/dev/stdout 2024-09-11T08:38:26.773 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:26.773 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/cluster1-0/keyring of=/dev/stdout 2024-09-11T08:38:26.826 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:26.826 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/cluster1-1/keyring of=/dev/stdout 2024-09-11T08:38:26.881 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:26.881 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/cluster1-2/keyring of=/dev/stdout 2024-09-11T08:38:26.934 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:26.934 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/cluster1.client.0.keyring of=/dev/stdout 2024-09-11T08:38:26.985 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:26.985 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.keyring of=/dev/stdout 2024-09-11T08:38:26.991 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:26.992 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.0.keyring of=/dev/stdout 2024-09-11T08:38:27.040 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:27.040 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.1.keyring of=/dev/stdout 2024-09-11T08:38:27.088 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:27.088 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.2.keyring of=/dev/stdout 2024-09-11T08:38:27.136 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:27.136 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.3.keyring of=/dev/stdout 2024-09-11T08:38:27.184 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:27.184 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.4.keyring of=/dev/stdout 2024-09-11T08:38:27.232 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:27.232 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.5.keyring of=/dev/stdout 2024-09-11T08:38:27.280 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:27.280 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster1.client.mirror.6.keyring of=/dev/stdout 2024-09-11T08:38:27.328 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T08:38:27.328 DEBUG:teuthology.orchestra.run.smithi073:> sudo tee -a /etc/ceph/cluster1.keyring 2024-09-11T08:38:27.340 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr.x] 2024-09-11T08:38:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD0VuFmdV5qLRAAVvWxN0N/QAKOaLe86pK7LQ== 2024-09-11T08:38:27.340 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.0] 2024-09-11T08:38:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD3VuFmApcBMBAAOSzlbF6STehnkQ8DE/hp+w== 2024-09-11T08:38:27.340 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.1] 2024-09-11T08:38:27.340 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD7VuFm3URhHxAAvs8kDSa+Ikdrsw9+qRkFEw== 2024-09-11T08:38:27.340 INFO:teuthology.orchestra.run.smithi073.stdout:[osd.2] 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD/VuFmm86mERAANA5tiFMtZpUMJs+9C/HZOQ== 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout:[client.0] 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD0VuFmdjBHMhAAlz1a++91UFg+yOZx7z2Imw== 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror] 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD0VuFmBQD7NBAAoj9BW40sjpRWncf16yvreA== 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.0] 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD1VuFmjU27ABAA3/f5R/9O5Xkf0dDZ28Efwg== 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.1] 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD1VuFmJQe+BRAA9YAzwwmEdvXnwEGRXvMYSg== 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.2] 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD1VuFms6qbChAAtYuBUZGXSvUb+867m857VQ== 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.3] 2024-09-11T08:38:27.341 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD1VuFmgBdKDxAAyYjRo1IlxwoKZ33G8zQ92g== 2024-09-11T08:38:27.342 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.4] 2024-09-11T08:38:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD1VuFmOYNBFBAAQLIQLiOEW/oTTZRUYOUTcw== 2024-09-11T08:38:27.342 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.5] 2024-09-11T08:38:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD1VuFm4ZVKGRAAdKWv7eQKvS9XEzZ/XZbxpA== 2024-09-11T08:38:27.342 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.6] 2024-09-11T08:38:27.342 INFO:teuthology.orchestra.run.smithi073.stdout: key = AQD1VuFmlkmMHhAApJkH4QWz34Ouf8UU4aXJeg== 2024-09-11T08:38:27.343 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T08:38:27.431 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T08:38:27.474 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T08:38:27.560 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T08:38:27.644 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:27.732 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:27.780 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:27.826 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:27.914 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:28.005 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:28.091 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:28.131 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:28.220 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster1.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:38:28.266 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T08:38:28.266 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mon/cluster1-a 2024-09-11T08:38:28.320 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster1 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster1.monmap --keyring /etc/ceph/cluster1.keyring 2024-09-11T08:38:28.403 DEBUG:teuthology.orchestra.run.smithi073:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster1-a 2024-09-11T08:38:28.456 DEBUG:teuthology.orchestra.run.smithi073:> rm -- /home/ubuntu/cephtest/cluster1.monmap 2024-09-11T08:38:28.502 INFO:tasks.ceph:Starting mon daemons in cluster cluster1... 2024-09-11T08:38:28.502 INFO:tasks.ceph.cluster1.mon.a:Restarting daemon 2024-09-11T08:38:28.502 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster1 -i a 2024-09-11T08:38:28.548 INFO:tasks.ceph.cluster1.mon.a:Started 2024-09-11T08:38:28.548 INFO:tasks.ceph:Starting mgr daemons in cluster cluster1... 2024-09-11T08:38:28.549 INFO:tasks.ceph.cluster1.mgr.x:Restarting daemon 2024-09-11T08:38:28.549 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster1 -i x 2024-09-11T08:38:28.551 INFO:tasks.ceph.cluster1.mgr.x:Started 2024-09-11T08:38:28.551 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T08:38:28.552 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster cluster1 osd crush tunables default 2024-09-11T08:38:28.814 INFO:teuthology.orchestra.run.smithi073.stderr:adjusted tunables profile to default 2024-09-11T08:38:28.832 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T08:38:28.832 INFO:tasks.ceph:Starting osd daemons in cluster cluster1... 2024-09-11T08:38:28.833 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:28.833 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/cluster1-0/fsid of=/dev/stdout 2024-09-11T08:38:28.845 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:28.845 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/cluster1-1/fsid of=/dev/stdout 2024-09-11T08:38:28.897 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:28.897 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/osd/cluster1-2/fsid of=/dev/stdout 2024-09-11T08:38:28.947 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:28.944+0000 7fc63b6ea180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:38:28.952 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster cluster1 osd new 9413ccf2-d186-4993-a666-0cb76bc6a459 0 2024-09-11T08:38:29.193 INFO:teuthology.orchestra.run.smithi073.stdout:0 2024-09-11T08:38:29.209 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster cluster1 osd new 58e98484-e7c9-412c-821d-5f3d8ea52733 1 2024-09-11T08:38:29.405 INFO:teuthology.orchestra.run.smithi073.stdout:1 2024-09-11T08:38:29.421 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster cluster1 osd new 4cf96554-e6a0-483a-aa70-66ac8a55f595 2 2024-09-11T08:38:29.479 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:29.476+0000 7fc63b6ea180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:38:29.620 INFO:teuthology.orchestra.run.smithi073.stdout:2 2024-09-11T08:38:29.636 INFO:tasks.ceph.cluster1.osd.0:Restarting daemon 2024-09-11T08:38:29.637 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 0 2024-09-11T08:38:29.638 INFO:tasks.ceph.cluster1.osd.0:Started 2024-09-11T08:38:29.639 INFO:tasks.ceph.cluster1.osd.1:Restarting daemon 2024-09-11T08:38:29.639 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 1 2024-09-11T08:38:29.641 INFO:tasks.ceph.cluster1.osd.1:Started 2024-09-11T08:38:29.641 INFO:tasks.ceph.cluster1.osd.2:Restarting daemon 2024-09-11T08:38:29.641 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster1 -i 2 2024-09-11T08:38:29.643 INFO:tasks.ceph.cluster1.osd.2:Started 2024-09-11T08:38:29.644 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T08:38:29.718 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:29.712+0000 7fc63b6ea180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:38:29.824 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:29.824 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":5,"fsid":"9661569a-ff40-4cdb-a5be-7b937214efd1","created":"2024-09-11T08:38:28.689051+0000","modified":"2024-09-11T08:38:29.621519+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T08:38:29.621519+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9413ccf2-d186-4993-a666-0cb76bc6a459","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"58e98484-e7c9-412c-821d-5f3d8ea52733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4cf96554-e6a0-483a-aa70-66ac8a55f595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:38:29.840 INFO:tasks.ceph.ceph_manager.cluster1:[] 2024-09-11T08:38:29.840 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T08:38:30.143 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T08:38:30.282 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.276+0000 7fc63b6ea180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:38:30.327 INFO:teuthology.misc.health.smithi073.stdout: 2024-09-11T08:38:30.327 INFO:teuthology.misc.health.smithi073.stdout:{"epoch":5,"fsid":"9661569a-ff40-4cdb-a5be-7b937214efd1","created":"2024-09-11T08:38:28.689051+0000","modified":"2024-09-11T08:38:29.621519+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T08:38:29.621519+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9413ccf2-d186-4993-a666-0cb76bc6a459","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"58e98484-e7c9-412c-821d-5f3d8ea52733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4cf96554-e6a0-483a-aa70-66ac8a55f595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:38:30.337 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.332+0000 7fc63b6ea180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:38:30.341 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T08:38:30.390 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.388+0000 7fc63b6ea180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:38:30.468 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.464+0000 7fc63b6ea180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:38:30.519 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.516+0000 7fc63b6ea180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:38:30.596 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.592+0000 7fc63b6ea180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:38:30.649 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.644+0000 7fc63b6ea180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:38:30.801 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.796+0000 7fc63b6ea180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:38:30.862 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:30.860+0000 7fc63b6ea180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:38:31.335 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T08:38:31.332+0000 7f658a3977c0 -1 Falling back to public interface 2024-09-11T08:38:31.516 INFO:tasks.ceph.cluster1.osd.1.smithi073.stderr:2024-09-11T08:38:31.512+0000 7fd1d7ce87c0 -1 Falling back to public interface 2024-09-11T08:38:31.535 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T08:38:31.535 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T08:38:31.535 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr: from numpy import show_config as show_numpy_config 2024-09-11T08:38:31.568 INFO:tasks.ceph.cluster1.osd.0.smithi073.stderr:2024-09-11T08:38:31.564+0000 7f5f586687c0 -1 Falling back to public interface 2024-09-11T08:38:31.589 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:31.584+0000 7fc63b6ea180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T08:38:31.746 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:31.744+0000 7fc63b6ea180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:38:31.894 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:31.888+0000 7fc63b6ea180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:38:31.957 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:31.952+0000 7fc63b6ea180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:38:32.092 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:32.088+0000 7fc63b6ea180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:38:32.702 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:32.696+0000 7fc63b6ea180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:38:32.777 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:32.772+0000 7fc63b6ea180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:38:32.960 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:32.956+0000 7fc63b6ea180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:38:33.033 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:33.028+0000 7fc63b6ea180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:38:33.326 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:33.320+0000 7fc63b6ea180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:38:33.388 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:33.384+0000 7fc63b6ea180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:38:34.624 INFO:tasks.ceph.cluster1.mgr.x.smithi073.stderr:2024-09-11T08:38:34.620+0000 7fc5e2220640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T08:38:36.107 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T08:38:36.104+0000 7f658a3977c0 -1 osd.2 0 log_to_monitors true 2024-09-11T08:38:36.216 INFO:tasks.ceph.cluster1.osd.1.smithi073.stderr:2024-09-11T08:38:36.212+0000 7fd1d7ce87c0 -1 osd.1 0 log_to_monitors true 2024-09-11T08:38:36.306 INFO:tasks.ceph.cluster1.osd.0.smithi073.stderr:2024-09-11T08:38:36.304+0000 7f5f586687c0 -1 osd.0 0 log_to_monitors true 2024-09-11T08:38:36.645 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T08:38:36.946 INFO:teuthology.misc.health.smithi073.stdout: 2024-09-11T08:38:36.946 INFO:teuthology.misc.health.smithi073.stdout:{"epoch":6,"fsid":"9661569a-ff40-4cdb-a5be-7b937214efd1","created":"2024-09-11T08:38:28.689051+0000","modified":"2024-09-11T08:38:36.464963+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T08:38:29.621519+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"9413ccf2-d186-4993-a666-0cb76bc6a459","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"58e98484-e7c9-412c-821d-5f3d8ea52733","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"4cf96554-e6a0-483a-aa70-66ac8a55f595","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:38:36.963 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T08:38:39.416 INFO:tasks.ceph.cluster1.osd.1.smithi073.stderr:2024-09-11T08:38:39.412+0000 7fd1d346e640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T08:38:39.428 INFO:tasks.ceph.cluster1.osd.0.smithi073.stderr:2024-09-11T08:38:39.424+0000 7f5f53dee640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T08:38:39.442 INFO:tasks.ceph.cluster1.osd.1.smithi073.stderr:2024-09-11T08:38:39.440+0000 7fd1cbab5640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:38:39.450 INFO:tasks.ceph.cluster1.osd.0.smithi073.stderr:2024-09-11T08:38:39.448+0000 7f5f4c435640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:38:39.464 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T08:38:39.460+0000 7f6585b1d640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T08:38:39.490 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T08:38:39.488+0000 7f657e164640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:38:43.267 DEBUG:teuthology.orchestra.run.smithi073:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster1 osd dump --format=json 2024-09-11T08:38:43.599 INFO:teuthology.misc.health.smithi073.stdout: 2024-09-11T08:38:43.599 INFO:teuthology.misc.health.smithi073.stdout:{"epoch":11,"fsid":"9661569a-ff40-4cdb-a5be-7b937214efd1","created":"2024-09-11T08:38:28.689051+0000","modified":"2024-09-11T08:38:43.502009+0000","last_up_change":"2024-09-11T08:38:40.431031+0000","last_in_change":"2024-09-11T08:38:29.621519+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:38:41.491826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9413ccf2-d186-4993-a666-0cb76bc6a459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6817","nonce":2968504120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6819","nonce":2968504120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6823","nonce":2968504120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6821","nonce":2968504120}]},"public_addr":"172.21.15.73:6817/2968504120","cluster_addr":"172.21.15.73:6819/2968504120","heartbeat_back_addr":"172.21.15.73:6823/2968504120","heartbeat_front_addr":"172.21.15.73:6821/2968504120","state":["exists","up"]},{"osd":1,"uuid":"58e98484-e7c9-412c-821d-5f3d8ea52733","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6809","nonce":56303921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6811","nonce":56303921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6815","nonce":56303921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6813","nonce":56303921}]},"public_addr":"172.21.15.73:6809/56303921","cluster_addr":"172.21.15.73:6811/56303921","heartbeat_back_addr":"172.21.15.73:6815/56303921","heartbeat_front_addr":"172.21.15.73:6813/56303921","state":["exists","up"]},{"osd":2,"uuid":"4cf96554-e6a0-483a-aa70-66ac8a55f595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6801","nonce":544452323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6803","nonce":544452323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6807","nonce":544452323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6805","nonce":544452323}]},"public_addr":"172.21.15.73:6801/544452323","cluster_addr":"172.21.15.73:6803/544452323","heartbeat_back_addr":"172.21.15.73:6807/544452323","heartbeat_front_addr":"172.21.15.73:6805/544452323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.267945+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.060900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:38:43.617 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T08:38:43.617 INFO:tasks.ceph:Creating RBD pool 2024-09-11T08:38:43.617 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster cluster1 osd pool create rbd 8 2024-09-11T08:38:44.579 INFO:teuthology.orchestra.run.smithi073.stderr:pool 'rbd' created 2024-09-11T08:38:44.607 DEBUG:teuthology.orchestra.run.smithi073:> rbd --cluster cluster1 pool init rbd 2024-09-11T08:38:47.650 INFO:tasks.ceph:Starting mds daemons in cluster cluster1... 2024-09-11T08:38:47.650 INFO:tasks.ceph:Waiting until cluster1 daemons up and pgs clean... 2024-09-11T08:38:47.651 INFO:tasks.ceph.ceph_manager.cluster1:waiting for mgr available 2024-09-11T08:38:47.651 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 mgr dump --format=json 2024-09-11T08:38:47.651 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T08:38:47.999 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:48.016 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":5,"flags":0,"active_gid":4099,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6824","nonce":1579791394},{"type":"v1","addr":"172.21.15.73:6825","nonce":1579791394}]},"active_addr":"172.21.15.73:6825/1579791394","active_change":"2024-09-11T08:38:33.409203+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":1040119710}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":3310307470}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":519530844}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":3351858896}]}]} 2024-09-11T08:38:48.018 INFO:tasks.ceph.ceph_manager.cluster1:mgr available! 2024-09-11T08:38:48.018 INFO:tasks.ceph.ceph_manager.cluster1:waiting for all up 2024-09-11T08:38:48.018 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T08:38:48.290 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:48.290 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":15,"fsid":"9661569a-ff40-4cdb-a5be-7b937214efd1","created":"2024-09-11T08:38:28.689051+0000","modified":"2024-09-11T08:38:47.601193+0000","last_up_change":"2024-09-11T08:38:40.431031+0000","last_in_change":"2024-09-11T08:38:29.621519+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:38:41.491826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T08:38:43.945430+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9413ccf2-d186-4993-a666-0cb76bc6a459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6817","nonce":2968504120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6819","nonce":2968504120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6823","nonce":2968504120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6821","nonce":2968504120}]},"public_addr":"172.21.15.73:6817/2968504120","cluster_addr":"172.21.15.73:6819/2968504120","heartbeat_back_addr":"172.21.15.73:6823/2968504120","heartbeat_front_addr":"172.21.15.73:6821/2968504120","state":["exists","up"]},{"osd":1,"uuid":"58e98484-e7c9-412c-821d-5f3d8ea52733","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6809","nonce":56303921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6811","nonce":56303921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6815","nonce":56303921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6813","nonce":56303921}]},"public_addr":"172.21.15.73:6809/56303921","cluster_addr":"172.21.15.73:6811/56303921","heartbeat_back_addr":"172.21.15.73:6815/56303921","heartbeat_front_addr":"172.21.15.73:6813/56303921","state":["exists","up"]},{"osd":2,"uuid":"4cf96554-e6a0-483a-aa70-66ac8a55f595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6801","nonce":544452323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6803","nonce":544452323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6807","nonce":544452323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6805","nonce":544452323}]},"public_addr":"172.21.15.73:6801/544452323","cluster_addr":"172.21.15.73:6803/544452323","heartbeat_back_addr":"172.21.15.73:6807/544452323","heartbeat_front_addr":"172.21.15.73:6805/544452323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.267945+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.060900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:38:48.307 INFO:tasks.ceph.ceph_manager.cluster1:all up! 2024-09-11T08:38:48.307 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T08:38:48.584 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:48.584 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":15,"fsid":"9661569a-ff40-4cdb-a5be-7b937214efd1","created":"2024-09-11T08:38:28.689051+0000","modified":"2024-09-11T08:38:47.601193+0000","last_up_change":"2024-09-11T08:38:40.431031+0000","last_in_change":"2024-09-11T08:38:29.621519+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:38:41.491826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T08:38:43.945430+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9413ccf2-d186-4993-a666-0cb76bc6a459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6817","nonce":2968504120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6819","nonce":2968504120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6823","nonce":2968504120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6821","nonce":2968504120}]},"public_addr":"172.21.15.73:6817/2968504120","cluster_addr":"172.21.15.73:6819/2968504120","heartbeat_back_addr":"172.21.15.73:6823/2968504120","heartbeat_front_addr":"172.21.15.73:6821/2968504120","state":["exists","up"]},{"osd":1,"uuid":"58e98484-e7c9-412c-821d-5f3d8ea52733","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6809","nonce":56303921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6811","nonce":56303921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6815","nonce":56303921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6813","nonce":56303921}]},"public_addr":"172.21.15.73:6809/56303921","cluster_addr":"172.21.15.73:6811/56303921","heartbeat_back_addr":"172.21.15.73:6815/56303921","heartbeat_front_addr":"172.21.15.73:6813/56303921","state":["exists","up"]},{"osd":2,"uuid":"4cf96554-e6a0-483a-aa70-66ac8a55f595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6801","nonce":544452323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6803","nonce":544452323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6807","nonce":544452323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6805","nonce":544452323}]},"public_addr":"172.21.15.73:6801/544452323","cluster_addr":"172.21.15.73:6803/544452323","heartbeat_back_addr":"172.21.15.73:6807/544452323","heartbeat_front_addr":"172.21.15.73:6805/544452323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.267945+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.060900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:38:48.602 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 flush_pg_stats 2024-09-11T08:38:48.602 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 flush_pg_stats 2024-09-11T08:38:48.603 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 flush_pg_stats 2024-09-11T08:38:48.754 INFO:teuthology.orchestra.run.smithi073.stdout:34359738373 2024-09-11T08:38:48.754 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-11T08:38:48.755 INFO:teuthology.orchestra.run.smithi073.stdout:34359738373 2024-09-11T08:38:48.756 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-11T08:38:48.763 INFO:teuthology.orchestra.run.smithi073.stdout:34359738373 2024-09-11T08:38:48.764 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-11T08:38:49.081 INFO:teuthology.orchestra.run.smithi073.stdout:34359738372 2024-09-11T08:38:49.085 INFO:teuthology.orchestra.run.smithi073.stdout:34359738372 2024-09-11T08:38:49.103 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.2 2024-09-11T08:38:49.104 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.0 2024-09-11T08:38:49.121 INFO:teuthology.orchestra.run.smithi073.stdout:34359738372 2024-09-11T08:38:49.137 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738372 for osd.1 2024-09-11T08:38:50.104 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.2 2024-09-11T08:38:50.105 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.0 2024-09-11T08:38:50.139 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd last-stat-seq osd.1 2024-09-11T08:38:50.396 INFO:teuthology.orchestra.run.smithi073.stdout:34359738373 2024-09-11T08:38:50.413 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.2 2024-09-11T08:38:50.414 DEBUG:teuthology.parallel:result is None 2024-09-11T08:38:50.415 INFO:teuthology.orchestra.run.smithi073.stdout:34359738373 2024-09-11T08:38:50.432 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.0 2024-09-11T08:38:50.432 DEBUG:teuthology.parallel:result is None 2024-09-11T08:38:50.435 INFO:teuthology.orchestra.run.smithi073.stdout:34359738373 2024-09-11T08:38:50.451 INFO:tasks.ceph.ceph_manager.cluster1:need seq 34359738373 got 34359738373 for osd.1 2024-09-11T08:38:50.451 DEBUG:teuthology.parallel:result is None 2024-09-11T08:38:50.452 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-11T08:38:50.452 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T08:38:50.781 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:50.782 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-09-11T08:38:50.797 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T08:38:49.419029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82636,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032884,"statfs":{"total":289910292480,"available":289825673216,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.826012"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.626418+0000","last_change":"2024-09-11T08:38:47.626867+0000","last_active":"2024-09-11T08:38:47.626418+0000","last_peered":"2024-09-11T08:38:47.626418+0000","last_clean":"2024-09-11T08:38:47.626418+0000","last_became_active":"2024-09-11T08:38:45.609230+0000","last_became_peered":"2024-09-11T08:38:45.609230+0000","last_unstale":"2024-09-11T08:38:47.626418+0000","last_undegraded":"2024-09-11T08:38:47.626418+0000","last_fullsized":"2024-09-11T08:38:47.626418+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:39:06.699918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021686650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625847+0000","last_change":"2024-09-11T08:38:47.626289+0000","last_active":"2024-09-11T08:38:47.625847+0000","last_peered":"2024-09-11T08:38:47.625847+0000","last_clean":"2024-09-11T08:38:47.625847+0000","last_became_active":"2024-09-11T08:38:45.609290+0000","last_became_peered":"2024-09-11T08:38:45.609290+0000","last_unstale":"2024-09-11T08:38:47.625847+0000","last_undegraded":"2024-09-11T08:38:47.625847+0000","last_fullsized":"2024-09-11T08:38:47.625847+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:54.913852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605272,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625212+0000","last_change":"2024-09-11T08:38:47.625598+0000","last_active":"2024-09-11T08:38:47.625212+0000","last_peered":"2024-09-11T08:38:47.625212+0000","last_clean":"2024-09-11T08:38:47.625212+0000","last_became_active":"2024-09-11T08:38:45.606786+0000","last_became_peered":"2024-09-11T08:38:45.606786+0000","last_unstale":"2024-09-11T08:38:47.625212+0000","last_undegraded":"2024-09-11T08:38:47.625212+0000","last_fullsized":"2024-09-11T08:38:47.625212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:01:02.286892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098543599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625068+0000","last_change":"2024-09-11T08:38:47.625546+0000","last_active":"2024-09-11T08:38:47.625068+0000","last_peered":"2024-09-11T08:38:47.625068+0000","last_clean":"2024-09-11T08:38:47.625068+0000","last_became_active":"2024-09-11T08:38:45.606856+0000","last_became_peered":"2024-09-11T08:38:45.606856+0000","last_unstale":"2024-09-11T08:38:47.625068+0000","last_undegraded":"2024-09-11T08:38:47.625068+0000","last_fullsized":"2024-09-11T08:38:47.625068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:04:07.641669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098068000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.622095+0000","last_change":"2024-09-11T08:38:47.622095+0000","last_active":"2024-09-11T08:38:47.622095+0000","last_peered":"2024-09-11T08:38:47.622095+0000","last_clean":"2024-09-11T08:38:47.622095+0000","last_became_active":"2024-09-11T08:38:45.603681+0000","last_became_peered":"2024-09-11T08:38:45.603681+0000","last_unstale":"2024-09-11T08:38:47.622095+0000","last_undegraded":"2024-09-11T08:38:47.622095+0000","last_fullsized":"2024-09-11T08:38:47.622095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:20:16.311770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021015780000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.614530+0000","last_change":"2024-09-11T08:38:47.615033+0000","last_active":"2024-09-11T08:38:47.614530+0000","last_peered":"2024-09-11T08:38:47.614530+0000","last_clean":"2024-09-11T08:38:47.614530+0000","last_became_active":"2024-09-11T08:38:45.604617+0000","last_became_peered":"2024-09-11T08:38:45.604617+0000","last_unstale":"2024-09-11T08:38:47.614530+0000","last_undegraded":"2024-09-11T08:38:47.614530+0000","last_fullsized":"2024-09-11T08:38:47.614530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:30.831517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010008059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.614451+0000","last_change":"2024-09-11T08:38:47.615034+0000","last_active":"2024-09-11T08:38:47.614451+0000","last_peered":"2024-09-11T08:38:47.614451+0000","last_clean":"2024-09-11T08:38:47.614451+0000","last_became_active":"2024-09-11T08:38:45.600541+0000","last_became_peered":"2024-09-11T08:38:45.600541+0000","last_unstale":"2024-09-11T08:38:47.614451+0000","last_undegraded":"2024-09-11T08:38:47.614451+0000","last_fullsized":"2024-09-11T08:38:47.614451+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:46.935084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625682+0000","last_change":"2024-09-11T08:38:47.626283+0000","last_active":"2024-09-11T08:38:47.625682+0000","last_peered":"2024-09-11T08:38:47.625682+0000","last_clean":"2024-09-11T08:38:47.625682+0000","last_became_active":"2024-09-11T08:38:45.605202+0000","last_became_peered":"2024-09-11T08:38:45.605202+0000","last_unstale":"2024-09-11T08:38:47.625682+0000","last_undegraded":"2024-09-11T08:38:47.625682+0000","last_fullsized":"2024-09-11T08:38:47.625682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:51:20.908060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016448249999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.624585+0000","last_change":"2024-09-11T08:38:43.315778+0000","last_active":"2024-09-11T08:38:47.624585+0000","last_peered":"2024-09-11T08:38:47.624585+0000","last_clean":"2024-09-11T08:38:47.624585+0000","last_became_active":"2024-09-11T08:38:43.314266+0000","last_became_peered":"2024-09-11T08:38:43.314266+0000","last_unstale":"2024-09-11T08:38:47.624585+0000","last_undegraded":"2024-09-11T08:38:47.624585+0000","last_fullsized":"2024-09-11T08:38:47.624585+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_clean_scrub_stamp":"2024-09-11T08:38:41.492511+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:19.438131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:38:50.798 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T08:38:51.062 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:51.062 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-09-11T08:38:51.079 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T08:38:49.419029+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82636,"kb_used_data":1552,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032884,"statfs":{"total":289910292480,"available":289825673216,"internally_reserved":0,"allocated":1589248,"data_stored":1295959,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":82963791},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"3.826012"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.626418+0000","last_change":"2024-09-11T08:38:47.626867+0000","last_active":"2024-09-11T08:38:47.626418+0000","last_peered":"2024-09-11T08:38:47.626418+0000","last_clean":"2024-09-11T08:38:47.626418+0000","last_became_active":"2024-09-11T08:38:45.609230+0000","last_became_peered":"2024-09-11T08:38:45.609230+0000","last_unstale":"2024-09-11T08:38:47.626418+0000","last_undegraded":"2024-09-11T08:38:47.626418+0000","last_fullsized":"2024-09-11T08:38:47.626418+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:39:06.699918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021686650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625847+0000","last_change":"2024-09-11T08:38:47.626289+0000","last_active":"2024-09-11T08:38:47.625847+0000","last_peered":"2024-09-11T08:38:47.625847+0000","last_clean":"2024-09-11T08:38:47.625847+0000","last_became_active":"2024-09-11T08:38:45.609290+0000","last_became_peered":"2024-09-11T08:38:45.609290+0000","last_unstale":"2024-09-11T08:38:47.625847+0000","last_undegraded":"2024-09-11T08:38:47.625847+0000","last_fullsized":"2024-09-11T08:38:47.625847+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:54.913852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605272,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625212+0000","last_change":"2024-09-11T08:38:47.625598+0000","last_active":"2024-09-11T08:38:47.625212+0000","last_peered":"2024-09-11T08:38:47.625212+0000","last_clean":"2024-09-11T08:38:47.625212+0000","last_became_active":"2024-09-11T08:38:45.606786+0000","last_became_peered":"2024-09-11T08:38:45.606786+0000","last_unstale":"2024-09-11T08:38:47.625212+0000","last_undegraded":"2024-09-11T08:38:47.625212+0000","last_fullsized":"2024-09-11T08:38:47.625212+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:01:02.286892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098543599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625068+0000","last_change":"2024-09-11T08:38:47.625546+0000","last_active":"2024-09-11T08:38:47.625068+0000","last_peered":"2024-09-11T08:38:47.625068+0000","last_clean":"2024-09-11T08:38:47.625068+0000","last_became_active":"2024-09-11T08:38:45.606856+0000","last_became_peered":"2024-09-11T08:38:45.606856+0000","last_unstale":"2024-09-11T08:38:47.625068+0000","last_undegraded":"2024-09-11T08:38:47.625068+0000","last_fullsized":"2024-09-11T08:38:47.625068+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:04:07.641669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098068000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.622095+0000","last_change":"2024-09-11T08:38:47.622095+0000","last_active":"2024-09-11T08:38:47.622095+0000","last_peered":"2024-09-11T08:38:47.622095+0000","last_clean":"2024-09-11T08:38:47.622095+0000","last_became_active":"2024-09-11T08:38:45.603681+0000","last_became_peered":"2024-09-11T08:38:45.603681+0000","last_unstale":"2024-09-11T08:38:47.622095+0000","last_undegraded":"2024-09-11T08:38:47.622095+0000","last_fullsized":"2024-09-11T08:38:47.622095+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:20:16.311770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021015780000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.614530+0000","last_change":"2024-09-11T08:38:47.615033+0000","last_active":"2024-09-11T08:38:47.614530+0000","last_peered":"2024-09-11T08:38:47.614530+0000","last_clean":"2024-09-11T08:38:47.614530+0000","last_became_active":"2024-09-11T08:38:45.604617+0000","last_became_peered":"2024-09-11T08:38:45.604617+0000","last_unstale":"2024-09-11T08:38:47.614530+0000","last_undegraded":"2024-09-11T08:38:47.614530+0000","last_fullsized":"2024-09-11T08:38:47.614530+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:30.831517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010008059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":17,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.614451+0000","last_change":"2024-09-11T08:38:47.615034+0000","last_active":"2024-09-11T08:38:47.614451+0000","last_peered":"2024-09-11T08:38:47.614451+0000","last_clean":"2024-09-11T08:38:47.614451+0000","last_became_active":"2024-09-11T08:38:45.600541+0000","last_became_peered":"2024-09-11T08:38:45.600541+0000","last_unstale":"2024-09-11T08:38:47.614451+0000","last_undegraded":"2024-09-11T08:38:47.614451+0000","last_fullsized":"2024-09-11T08:38:47.614451+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:46.935084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.625682+0000","last_change":"2024-09-11T08:38:47.626283+0000","last_active":"2024-09-11T08:38:47.625682+0000","last_peered":"2024-09-11T08:38:47.625682+0000","last_clean":"2024-09-11T08:38:47.625682+0000","last_became_active":"2024-09-11T08:38:45.605202+0000","last_became_peered":"2024-09-11T08:38:45.605202+0000","last_unstale":"2024-09-11T08:38:47.625682+0000","last_undegraded":"2024-09-11T08:38:47.625682+0000","last_fullsized":"2024-09-11T08:38:47.625682+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:51:20.908060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016448249999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"11'97","reported_seq":147,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:38:47.624585+0000","last_change":"2024-09-11T08:38:43.315778+0000","last_active":"2024-09-11T08:38:47.624585+0000","last_peered":"2024-09-11T08:38:47.624585+0000","last_clean":"2024-09-11T08:38:47.624585+0000","last_became_active":"2024-09-11T08:38:43.314266+0000","last_became_peered":"2024-09-11T08:38:43.314266+0000","last_unstale":"2024-09-11T08:38:47.624585+0000","last_undegraded":"2024-09-11T08:38:47.624585+0000","last_fullsized":"2024-09-11T08:38:47.624585+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_clean_scrub_stamp":"2024-09-11T08:38:41.492511+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:19.438131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":2,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27156,"kb_used_data":128,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344684,"statfs":{"total":96636764160,"available":96608956416,"internally_reserved":0,"allocated":131072,"data_stored":38395,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738373,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:38:51.080 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-11T08:38:51.080 INFO:tasks.ceph:Waiting until ceph cluster cluster1 is healthy... 2024-09-11T08:38:51.080 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy 2024-09-11T08:38:51.080 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 health --format=json 2024-09-11T08:38:51.399 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:51.399 INFO:teuthology.orchestra.run.smithi073.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T08:38:51.416 INFO:tasks.ceph.ceph_manager.cluster1:wait_until_healthy done 2024-09-11T08:38:51.416 INFO:teuthology.run_tasks:Running task ceph... 2024-09-11T08:38:51.429 INFO:tasks.ceph:Creating ceph cluster cluster2... 2024-09-11T08:38:51.430 INFO:tasks.ceph:config {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'cpu_profile': set(), 'cluster': 'cluster2', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2024-09-11T08:38:51.430 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659', 'branch': 'squid', 'description': 'rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock}', 'email': 'ceph-qa@ceph.com', 'first_in_suite': False, 'job_id': '7900659', 'kernel': {'kdb': 1, 'sha1': 'distro'}, 'ktype': 'distro', 'last_in_suite': False, 'machine_type': 'smithi', 'meta': [{'desc': 'run two ceph clusters and install rbd-mirror'}, {'desc': 'configure the permissions for client.mirror'}, {'desc': '2 ceph clusters with 1 mon and 3 osds each'}, {'desc': 'run four rbd-mirror daemons per cluster'}, {'desc': 'run the rbd_mirror_stress.sh workunit to test the rbd-mirror daemon'}], 'name': 'teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi', 'no_nested_subset': False, 'openstack': [{'volumes': {'count': 3, 'size': 30}}], 'os_type': 'ubuntu', 'os_version': '22.04', 'overrides': {'admin_socket': {'branch': 'squid'}, 'ceph': {'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {}}}, 'install': {'ceph': {'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, 'thrashosds': {'bdev_inject_crash': 2, 'bdev_inject_crash_probability': 0.5}, 'workunit': {'branch': 'squid', 'sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae'}}, 'owner': 'scheduled_teuthology@teuthology', 'priority': 100, 'repo': 'https://git.ceph.com/ceph.git', 'roles': [['cluster1.mon.a', 'cluster1.mgr.x', 'cluster2.mgr.x', 'cluster1.osd.0', 'cluster1.osd.1', 'cluster1.osd.2', 'cluster1.client.0', 'cluster2.client.0'], ['cluster2.mon.a', 'cluster2.osd.0', 'cluster2.osd.1', 'cluster2.osd.2', 'cluster1.client.mirror', 'cluster1.client.mirror.0', 'cluster1.client.mirror.1', 'cluster1.client.mirror.2', 'cluster1.client.mirror.3', 'cluster1.client.mirror.4', 'cluster1.client.mirror.5', 'cluster1.client.mirror.6', 'cluster2.client.mirror', 'cluster2.client.mirror.0', 'cluster2.client.mirror.1', 'cluster2.client.mirror.2', 'cluster2.client.mirror.3', 'cluster2.client.mirror.4', 'cluster2.client.mirror.5', 'cluster2.client.mirror.6']], 'seed': 6175, 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc', 'sleep_before_teardown': 0, 'subset': '88/128', 'suite': 'rbd', 'suite_branch': 'squid', 'suite_path': '/home/teuthworker/src/git.ceph.com_ceph_daaf78de334647fc0c800f4ac495cc9475a26bae/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://git.ceph.com/ceph.git', 'suite_sha1': 'daaf78de334647fc0c800f4ac495cc9475a26bae', 'targets': {'smithi073.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBCuh0QeHx3fryzEpQpK5xbTHvxwgQg32edZPQTXOSJGseSUdnXvrij6JDuffVEHtSPdwy3PgxAkZctvxRi6AXoo=', 'smithi077.front.sepia.ceph.com': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOZ2WuzHB7zjUUpe0P9YBQNHv5XrPIF9koGEMtfiHnIw/SlLQrufkVBxayqxTGeezl22EHKhHzsda1D5Jb1zis4='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': 1, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': ['rbd-mirror'], 'flavor': 'default', 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster1', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'ceph': {'cluster': 'cluster2', 'conf': {'client': {'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'global': {'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(OSD_SLOW_PING_TIME'], 'sha1': 'b53b6d3e9f4baeb64d08dfb72b9cb1112ce886fc'}}, {'exec': {'cluster1.client.mirror.0': ["sudo ceph --cluster cluster1 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster1 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror mon 'profile rbd-mirror-peer' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.0 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.1 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.2 mon 'profile rbd-mirror' osd 'profile rbd'", "sudo ceph --cluster cluster2 auth caps client.mirror.3 mon 'profile rbd-mirror' osd 'profile rbd'"]}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster1.client.mirror.3', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.0', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.1', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.2', 'thrash': True}}, {'rbd-mirror': {'client': 'cluster2.client.mirror.3', 'thrash': True}}, {'rbd-mirror-thrash': {'cluster': 'cluster1'}}, {'rbd-mirror-thrash': {'cluster': 'cluster2'}}, {'workunit': {'clients': {'cluster1.client.mirror': ['rbd/rbd_mirror_stress.sh']}, 'env': {'CEPH_ARGS': '', 'MIRROR_POOL_MODE': 'image', 'RBD_IMAGE_FEATURES': 'layering,exclusive-lock', 'RBD_MIRROR_INSTANCES': '4', 'RBD_MIRROR_MODE': 'snapshot', 'RBD_MIRROR_USE_EXISTING_CLUSTER': '1', 'RBD_MIRROR_USE_RBD_MIRROR': '1'}, 'timeout': '6h'}}], 'teuthology': {'fragments_dropped': [], 'meta': {}, 'postmerge': []}, 'teuthology_branch': 'main', 'teuthology_sha1': '3752d3834a7b6cd13dc17dfaa6c2fd3658f3439a', 'timestamp': '2024-09-10_21:16:03', 'tube': 'smithi', 'user': 'teuthology', 'verbose': False, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.3401856'} 2024-09-11T08:38:51.430 DEBUG:teuthology.orchestra.run.smithi073:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-11T08:38:51.432 DEBUG:teuthology.orchestra.run.smithi077:> install -d -m0755 -- /home/ubuntu/cephtest/cluster2.data 2024-09-11T08:38:51.438 DEBUG:teuthology.orchestra.run.smithi073:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T08:38:51.479 DEBUG:teuthology.orchestra.run.smithi077:> sudo install -d -m0777 -- /var/run/ceph 2024-09-11T08:38:51.496 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:51.496 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/scratch_devs of=/dev/stdout 2024-09-11T08:38:51.543 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-09-11T08:38:51.544 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_1 2024-09-11T08:38:51.592 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-09-11T08:38:51.592 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:51.592 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 869 Links: 1 2024-09-11T08:38:51.592 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:51.592 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-09-11 08:32:50.552088627 +0000 2024-09-11T08:38:51.592 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-09-11 08:32:50.296090914 +0000 2024-09-11T08:38:51.592 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-09-11 08:32:50.296090914 +0000 2024-09-11T08:38:51.593 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-09-11T08:38:51.593 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-09-11T08:38:51.646 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-09-11T08:38:51.646 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-09-11T08:38:51.646 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000447266 s, 1.1 MB/s 2024-09-11T08:38:51.648 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-09-11T08:38:51.697 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_2 2024-09-11T08:38:51.743 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-09-11T08:38:51.743 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:51.743 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 886 Links: 1 2024-09-11T08:38:51.743 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:51.744 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-09-11 08:32:50.788086519 +0000 2024-09-11T08:38:51.744 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-09-11 08:32:50.784086555 +0000 2024-09-11T08:38:51.744 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-09-11 08:32:50.784086555 +0000 2024-09-11T08:38:51.744 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-09-11T08:38:51.744 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-09-11T08:38:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-09-11T08:38:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-09-11T08:38:51.798 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000427549 s, 1.2 MB/s 2024-09-11T08:38:51.799 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-09-11T08:38:51.850 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_3 2024-09-11T08:38:51.895 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-09-11T08:38:51.895 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:51.896 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 903 Links: 1 2024-09-11T08:38:51.896 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:51.896 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-09-11 08:32:51.508080087 +0000 2024-09-11T08:38:51.896 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-09-11 08:32:51.288082053 +0000 2024-09-11T08:38:51.896 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-09-11 08:32:51.288082053 +0000 2024-09-11T08:38:51.896 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-09-11T08:38:51.896 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-09-11T08:38:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-09-11T08:38:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-09-11T08:38:51.950 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000405024 s, 1.3 MB/s 2024-09-11T08:38:51.951 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-09-11T08:38:52.001 DEBUG:teuthology.orchestra.run.smithi077:> stat /dev/vg_nvme/lv_4 2024-09-11T08:38:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-09-11T08:38:52.047 INFO:teuthology.orchestra.run.smithi077.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-09-11T08:38:52.048 INFO:teuthology.orchestra.run.smithi077.stdout:Device: 5h/5d Inode: 926 Links: 1 2024-09-11T08:38:52.048 INFO:teuthology.orchestra.run.smithi077.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-09-11T08:38:52.048 INFO:teuthology.orchestra.run.smithi077.stdout:Access: 2024-09-11 08:32:52.008075622 +0000 2024-09-11T08:38:52.048 INFO:teuthology.orchestra.run.smithi077.stdout:Modify: 2024-09-11 08:32:51.740078015 +0000 2024-09-11T08:38:52.048 INFO:teuthology.orchestra.run.smithi077.stdout:Change: 2024-09-11 08:32:51.740078015 +0000 2024-09-11T08:38:52.048 INFO:teuthology.orchestra.run.smithi077.stdout: Birth: - 2024-09-11T08:38:52.048 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-09-11T08:38:52.102 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records in 2024-09-11T08:38:52.103 INFO:teuthology.orchestra.run.smithi077.stderr:1+0 records out 2024-09-11T08:38:52.103 INFO:teuthology.orchestra.run.smithi077.stderr:512 bytes copied, 0.000327356 s, 1.6 MB/s 2024-09-11T08:38:52.104 DEBUG:teuthology.orchestra.run.smithi077:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-09-11T08:38:52.153 INFO:tasks.ceph:osd dev map: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:52.153 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T08:38:52.154 INFO:tasks.ceph:Generating config... 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client] debug rbd = 20 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client] debug rbd_mirror = 15 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client] log to stderr = False 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client] rbd default features = 125 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client] rbd mirror image policy type = simple 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client.mirror.0] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client.mirror.0] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client.mirror.1] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client.mirror.1] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client.mirror.2] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.155 INFO:tasks.ceph:[client.mirror.2] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.156 INFO:tasks.ceph:[client.mirror.3] admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.156 INFO:tasks.ceph:[client.mirror.3] pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.156 INFO:tasks.ceph:[global] mon client directed command retry = 5 2024-09-11T08:38:52.156 INFO:tasks.ceph:[global] mon warn on pool no app = False 2024-09-11T08:38:52.156 INFO:tasks.ceph:[global] ms inject socket failures = 5000 2024-09-11T08:38:52.156 INFO:tasks.ceph:[mgr] debug mgr = 20 2024-09-11T08:38:52.156 INFO:tasks.ceph:[mgr] debug ms = 1 2024-09-11T08:38:52.156 INFO:tasks.ceph:[mon] debug mon = 20 2024-09-11T08:38:52.156 INFO:tasks.ceph:[mon] debug ms = 1 2024-09-11T08:38:52.156 INFO:tasks.ceph:[mon] debug paxos = 20 2024-09-11T08:38:52.156 INFO:tasks.ceph:[osd] bdev async discard = True 2024-09-11T08:38:52.156 INFO:tasks.ceph:[osd] bdev enable discard = True 2024-09-11T08:38:52.156 INFO:tasks.ceph:[osd] bluestore block size = 96636764160 2024-09-11T08:38:52.156 INFO:tasks.ceph:[osd] bluestore fsck on mount = True 2024-09-11T08:38:52.156 INFO:tasks.ceph:[osd] debug bluefs = 1/20 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] debug bluestore = 1/20 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] debug ms = 1 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] debug osd = 20 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] debug rocksdb = 4/10 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] mon osd backfillfull_ratio = 0.85 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] mon osd full ratio = 0.9 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] mon osd nearfull ratio = 0.8 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] osd failsafe full ratio = 0.95 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] osd memory target = 1610612736 2024-09-11T08:38:52.157 INFO:tasks.ceph:[osd] osd objectstore = bluestore 2024-09-11T08:38:52.157 INFO:tasks.ceph:Setting up cluster2.mon.a... 2024-09-11T08:38:52.157 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/cluster2.keyring 2024-09-11T08:38:52.226 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.keyring 2024-09-11T08:38:52.229 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/cluster2.keyring 2024-09-11T08:38:52.305 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T08:38:52.359 DEBUG:tasks.ceph:Ceph mon addresses: [('cluster2.mon.a', '172.21.15.77')] 2024-09-11T08:38:52.359 DEBUG:tasks.ceph:writing out conf {'global': {'chdir': '', 'pid file': '/var/run/ceph/$cluster-$name.pid', 'auth supported': 'cephx', 'filestore xattr use omap': 'true', 'mon clock drift allowed': '1.000', 'osd crush chooseleaf type': '0', 'auth debug': 'true', 'ms die on old message': 'true', 'ms die on bug': 'true', 'mon max pg per osd': '10000', 'mon pg warn max object skew': '0', 'osd_pool_default_pg_autoscale_mode': 'off', 'osd pool default size': '2', 'mon osd allow primary affinity': 'true', 'mon osd allow pg remap': 'true', 'mon warn on legacy crush tunables': 'false', 'mon warn on crush straw calc version zero': 'false', 'mon warn on no sortbitwise': 'false', 'mon warn on osd down out interval zero': 'false', 'mon warn on too few osds': 'false', 'mon_warn_on_pool_pg_num_not_power_of_two': 'false', 'mon_warn_on_pool_no_redundancy': 'false', 'mon_allow_pool_size_one': 'true', 'osd pool default erasure code profile': 'plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd', 'osd default data pool replay window': '5', 'mon allow pool delete': 'true', 'mon cluster log file level': 'debug', 'debug asserts on shutdown': 'true', 'mon health detail to clog': 'false', 'mon host': '172.21.15.77', 'mon client directed command retry': 5, 'mon warn on pool no app': False, 'ms inject socket failures': 5000}, 'osd': {'osd journal size': '100', 'osd scrub load threshold': '5.0', 'osd scrub max interval': '600', 'osd mclock profile': 'high_recovery_ops', 'osd recover clone overlap': 'true', 'osd recovery max chunk': '1048576', 'osd debug shutdown': 'true', 'osd debug op order': 'true', 'osd debug verify stray on activate': 'true', 'osd debug trim objects': 'true', 'osd open classes on start': 'true', 'osd debug pg log writeout': 'true', 'osd deep scrub update digest min age': '30', 'osd map max advance': '10', 'journal zero on create': 'true', 'filestore ondisk finisher threads': '3', 'filestore apply finisher threads': '3', 'bdev debug aio': 'true', 'osd debug misdirected ops': 'true', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd memory target': 1610612736, 'osd objectstore': 'bluestore'}, 'mgr': {'debug ms': 1, 'debug mgr': 20, 'debug mon': '20', 'debug auth': '20', 'mon reweight min pgs per osd': '4', 'mon reweight min bytes per osd': '10', 'mgr/telemetry/nag': 'false'}, 'mon': {'debug ms': 1, 'debug mon': 20, 'debug paxos': 20, 'debug auth': '20', 'mon data avail warn': '5', 'mon mgr mkfs grace': '240', 'mon reweight min pgs per osd': '4', 'mon osd reporter subtree level': 'osd', 'mon osd prime pg temp': 'true', 'mon reweight min bytes per osd': '10', 'auth mon ticket ttl': '660', 'auth service ticket ttl': '240', 'mon_warn_on_insecure_global_id_reclaim': 'false', 'mon_warn_on_insecure_global_id_reclaim_allowed': 'false', 'mon_down_mkfs_grace': '2m', 'mon_warn_on_filestore_osds': 'false'}, 'client': {'rgw cache enabled': 'true', 'rgw enable ops log': 'true', 'rgw enable usage log': 'true', 'log file': '/var/log/ceph/$cluster-$name.$pid.log', 'admin socket': '/var/run/ceph/$cluster-$name.$pid.asok', 'debug rbd': 20, 'debug rbd_mirror': 15, 'log to stderr': False, 'rbd default features': 125, 'rbd mirror image policy type': 'simple'}, 'mon.a': {}, 'client.mirror.0': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.1': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.2': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}, 'client.mirror.3': {'admin socket': '/var/run/ceph/rbd-mirror.$cluster-$name.asok', 'pid file': '/var/run/ceph/rbd-mirror.$cluster-$name.pid'}} 2024-09-11T08:38:52.360 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:52.361 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T08:38:52.409 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool -c /home/ubuntu/cephtest/ceph.tmp.conf --create --clobber --enable-all-features --add a 172.21.15.77 --print /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:38:52.487 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: monmap file /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:38:52.487 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: generated fsid 600c207a-6d49-48ae-9606-243c4786b265 2024-09-11T08:38:52.488 INFO:teuthology.orchestra.run.smithi077.stdout:setting min_mon_release = quincy 2024-09-11T08:38:52.488 INFO:teuthology.orchestra.run.smithi077.stdout:epoch 0 2024-09-11T08:38:52.488 INFO:teuthology.orchestra.run.smithi077.stdout:fsid 600c207a-6d49-48ae-9606-243c4786b265 2024-09-11T08:38:52.488 INFO:teuthology.orchestra.run.smithi077.stdout:last_changed 2024-09-11T08:38:52.488544+0000 2024-09-11T08:38:52.488 INFO:teuthology.orchestra.run.smithi077.stdout:created 2024-09-11T08:38:52.488544+0000 2024-09-11T08:38:52.489 INFO:teuthology.orchestra.run.smithi077.stdout:min_mon_release 17 (quincy) 2024-09-11T08:38:52.489 INFO:teuthology.orchestra.run.smithi077.stdout:election_strategy: 1 2024-09-11T08:38:52.489 INFO:teuthology.orchestra.run.smithi077.stdout:0: [v2:172.21.15.77:3300/0,v1:172.21.15.77:6789/0] mon.a 2024-09-11T08:38:52.489 INFO:teuthology.orchestra.run.smithi077.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/cluster2.monmap (1 monitors) 2024-09-11T08:38:52.491 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/ceph.tmp.conf 2024-09-11T08:38:52.540 INFO:tasks.ceph:Writing /etc/ceph/cluster2.conf for FSID 600c207a-6d49-48ae-9606-243c4786b265... 2024-09-11T08:38:52.541 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-11T08:38:52.543 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/cluster2.conf && sudo chmod 0644 /etc/ceph/cluster2.conf > /dev/null 2024-09-11T08:38:52.587 INFO:teuthology.orchestra.run.smithi073.stdout:[global] 2024-09-11T08:38:52.587 INFO:teuthology.orchestra.run.smithi073.stdout: chdir = "" 2024-09-11T08:38:52.587 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T08:38:52.587 INFO:teuthology.orchestra.run.smithi073.stdout: auth supported = cephx 2024-09-11T08:38:52.588 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.588 INFO:teuthology.orchestra.run.smithi073.stdout: filestore xattr use omap = true 2024-09-11T08:38:52.588 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.588 INFO:teuthology.orchestra.run.smithi073.stdout: mon clock drift allowed = 1.000 2024-09-11T08:38:52.588 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.588 INFO:teuthology.orchestra.run.smithi073.stdout: osd crush chooseleaf type = 0 2024-09-11T08:38:52.589 INFO:teuthology.orchestra.run.smithi073.stdout: auth debug = true 2024-09-11T08:38:52.589 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.589 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on old message = true 2024-09-11T08:38:52.589 INFO:teuthology.orchestra.run.smithi073.stdout: ms die on bug = true 2024-09-11T08:38:52.589 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.589 INFO:teuthology.orchestra.run.smithi073.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T08:38:52.590 INFO:teuthology.orchestra.run.smithi073.stdout: mon pg warn max object skew = 0 2024-09-11T08:38:52.590 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.590 INFO:teuthology.orchestra.run.smithi073.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T08:38:52.590 INFO:teuthology.orchestra.run.smithi073.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T08:38:52.590 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.590 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default size = 2 2024-09-11T08:38:52.591 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.591 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow primary affinity = true 2024-09-11T08:38:52.591 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd allow pg remap = true 2024-09-11T08:38:52.591 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on legacy crush tunables = false 2024-09-11T08:38:52.591 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on crush straw calc version zero = false 2024-09-11T08:38:52.592 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on no sortbitwise = false 2024-09-11T08:38:52.592 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on osd down out interval zero = false 2024-09-11T08:38:52.592 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on too few osds = false 2024-09-11T08:38:52.592 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T08:38:52.592 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T08:38:52.592 INFO:teuthology.orchestra.run.smithi073.stdout: mon_allow_pool_size_one = true 2024-09-11T08:38:52.593 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.593 INFO:teuthology.orchestra.run.smithi073.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T08:38:52.593 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.593 INFO:teuthology.orchestra.run.smithi073.stdout: osd default data pool replay window = 5 2024-09-11T08:38:52.593 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.594 INFO:teuthology.orchestra.run.smithi073.stdout: mon allow pool delete = true 2024-09-11T08:38:52.594 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.594 INFO:teuthology.orchestra.run.smithi073.stdout: mon cluster log file level = debug 2024-09-11T08:38:52.594 INFO:teuthology.orchestra.run.smithi073.stdout: debug asserts on shutdown = true 2024-09-11T08:38:52.594 INFO:teuthology.orchestra.run.smithi073.stdout: mon health detail to clog = false 2024-09-11T08:38:52.594 INFO:teuthology.orchestra.run.smithi073.stdout: mon host = 172.21.15.77 2024-09-11T08:38:52.595 INFO:teuthology.orchestra.run.smithi073.stdout: mon client directed command retry = 5 2024-09-11T08:38:52.595 INFO:teuthology.orchestra.run.smithi073.stdout: mon warn on pool no app = False 2024-09-11T08:38:52.595 INFO:teuthology.orchestra.run.smithi073.stdout: ms inject socket failures = 5000 2024-09-11T08:38:52.595 INFO:teuthology.orchestra.run.smithi073.stdout: fsid = 600c207a-6d49-48ae-9606-243c4786b265 2024-09-11T08:38:52.595 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.596 INFO:teuthology.orchestra.run.smithi073.stdout:[osd] 2024-09-11T08:38:52.596 INFO:teuthology.orchestra.run.smithi073.stdout: osd journal size = 100 2024-09-11T08:38:52.596 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.596 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub load threshold = 5.0 2024-09-11T08:38:52.596 INFO:teuthology.orchestra.run.smithi073.stdout: osd scrub max interval = 600 2024-09-11T08:38:52.596 INFO:teuthology.orchestra.run.smithi073.stdout: osd mclock profile = high_recovery_ops 2024-09-11T08:38:52.597 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.597 INFO:teuthology.orchestra.run.smithi073.stdout: osd recover clone overlap = true 2024-09-11T08:38:52.597 INFO:teuthology.orchestra.run.smithi073.stdout: osd recovery max chunk = 1048576 2024-09-11T08:38:52.597 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.597 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug shutdown = true 2024-09-11T08:38:52.597 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug op order = true 2024-09-11T08:38:52.598 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug verify stray on activate = true 2024-09-11T08:38:52.598 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug trim objects = true 2024-09-11T08:38:52.598 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.598 INFO:teuthology.orchestra.run.smithi073.stdout: osd open classes on start = true 2024-09-11T08:38:52.598 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug pg log writeout = true 2024-09-11T08:38:52.598 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.599 INFO:teuthology.orchestra.run.smithi073.stdout: osd deep scrub update digest min age = 30 2024-09-11T08:38:52.599 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.599 INFO:teuthology.orchestra.run.smithi073.stdout: osd map max advance = 10 2024-09-11T08:38:52.599 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.599 INFO:teuthology.orchestra.run.smithi073.stdout: journal zero on create = true 2024-09-11T08:38:52.600 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.600 INFO:teuthology.orchestra.run.smithi073.stdout: filestore ondisk finisher threads = 3 2024-09-11T08:38:52.600 INFO:teuthology.orchestra.run.smithi073.stdout: filestore apply finisher threads = 3 2024-09-11T08:38:52.600 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.600 INFO:teuthology.orchestra.run.smithi073.stdout: bdev debug aio = true 2024-09-11T08:38:52.600 INFO:teuthology.orchestra.run.smithi073.stdout: osd debug misdirected ops = true 2024-09-11T08:38:52.601 INFO:teuthology.orchestra.run.smithi073.stdout: bdev async discard = True 2024-09-11T08:38:52.601 INFO:teuthology.orchestra.run.smithi073.stdout: bdev enable discard = True 2024-09-11T08:38:52.601 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore block size = 96636764160 2024-09-11T08:38:52.601 INFO:teuthology.orchestra.run.smithi073.stdout: bluestore fsck on mount = True 2024-09-11T08:38:52.601 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluefs = 1/20 2024-09-11T08:38:52.601 INFO:teuthology.orchestra.run.smithi073.stdout: debug bluestore = 1/20 2024-09-11T08:38:52.601 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-09-11T08:38:52.602 INFO:teuthology.orchestra.run.smithi073.stdout: debug osd = 20 2024-09-11T08:38:52.602 INFO:teuthology.orchestra.run.smithi073.stdout: debug rocksdb = 4/10 2024-09-11T08:38:52.602 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T08:38:52.602 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd full ratio = 0.9 2024-09-11T08:38:52.602 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd nearfull ratio = 0.8 2024-09-11T08:38:52.603 INFO:teuthology.orchestra.run.smithi073.stdout: osd failsafe full ratio = 0.95 2024-09-11T08:38:52.603 INFO:teuthology.orchestra.run.smithi073.stdout: osd memory target = 1610612736 2024-09-11T08:38:52.603 INFO:teuthology.orchestra.run.smithi073.stdout: osd objectstore = bluestore 2024-09-11T08:38:52.603 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.603 INFO:teuthology.orchestra.run.smithi073.stdout:[mgr] 2024-09-11T08:38:52.603 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-09-11T08:38:52.604 INFO:teuthology.orchestra.run.smithi073.stdout: debug mgr = 20 2024-09-11T08:38:52.604 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2024-09-11T08:38:52.604 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2024-09-11T08:38:52.604 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:52.604 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:52.604 INFO:teuthology.orchestra.run.smithi073.stdout: mgr/telemetry/nag = false 2024-09-11T08:38:52.604 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout:[mon] 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: debug ms = 1 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: debug mon = 20 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: debug paxos = 20 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: debug auth = 20 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: mon data avail warn = 5 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: mon mgr mkfs grace = 240 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:52.605 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd reporter subtree level = osd 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: mon osd prime pg temp = true 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: # 1m isn't quite enough 2024-09-11T08:38:52.606 INFO:teuthology.orchestra.run.smithi073.stdout: mon_down_mkfs_grace = 2m 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: mon_warn_on_filestore_osds = false 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout:[client] 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: rgw cache enabled = true 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable ops log = true 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: rgw enable usage log = true 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: debug rbd = 20 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: debug rbd_mirror = 15 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: log to stderr = False 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: rbd default features = 125 2024-09-11T08:38:52.607 INFO:teuthology.orchestra.run.smithi073.stdout: rbd mirror image policy type = simple 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout:[mon.a] 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.0] 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.1] 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.2] 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout:[client.mirror.3] 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.608 INFO:teuthology.orchestra.run.smithi073.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.614 INFO:teuthology.orchestra.run.smithi077.stdout:[global] 2024-09-11T08:38:52.614 INFO:teuthology.orchestra.run.smithi077.stdout: chdir = "" 2024-09-11T08:38:52.614 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2024-09-11T08:38:52.614 INFO:teuthology.orchestra.run.smithi077.stdout: auth supported = cephx 2024-09-11T08:38:52.614 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: filestore xattr use omap = true 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: mon clock drift allowed = 1.000 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: osd crush chooseleaf type = 0 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: auth debug = true 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on old message = true 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: ms die on bug = true 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: mon max pg per osd = 10000 # >= luminous 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: mon pg warn max object skew = 0 2024-09-11T08:38:52.615 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: # disable pg_autoscaler by default for new pools 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: osd_pool_default_pg_autoscale_mode = off 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default size = 2 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow primary affinity = true 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd allow pg remap = true 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on legacy crush tunables = false 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on crush straw calc version zero = false 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on no sortbitwise = false 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on osd down out interval zero = false 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on too few osds = false 2024-09-11T08:38:52.616 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_pool_no_redundancy = false 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: mon_allow_pool_size_one = true 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: osd default data pool replay window = 5 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: mon allow pool delete = true 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: mon cluster log file level = debug 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: debug asserts on shutdown = true 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: mon health detail to clog = false 2024-09-11T08:38:52.617 INFO:teuthology.orchestra.run.smithi077.stdout: mon host = 172.21.15.77 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: mon client directed command retry = 5 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: mon warn on pool no app = False 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: ms inject socket failures = 5000 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: fsid = 600c207a-6d49-48ae-9606-243c4786b265 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout:[osd] 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: osd journal size = 100 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub load threshold = 5.0 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: osd scrub max interval = 600 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: osd mclock profile = high_recovery_ops 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.618 INFO:teuthology.orchestra.run.smithi077.stdout: osd recover clone overlap = true 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd recovery max chunk = 1048576 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug shutdown = true 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug op order = true 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug verify stray on activate = true 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug trim objects = true 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd open classes on start = true 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug pg log writeout = true 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: osd deep scrub update digest min age = 30 2024-09-11T08:38:52.619 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: osd map max advance = 10 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: journal zero on create = true 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: filestore ondisk finisher threads = 3 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: filestore apply finisher threads = 3 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: bdev debug aio = true 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: osd debug misdirected ops = true 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: bdev async discard = True 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: bdev enable discard = True 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore block size = 96636764160 2024-09-11T08:38:52.620 INFO:teuthology.orchestra.run.smithi077.stdout: bluestore fsck on mount = True 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluefs = 1/20 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: debug bluestore = 1/20 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: debug osd = 20 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: debug rocksdb = 4/10 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd backfillfull_ratio = 0.85 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd full ratio = 0.9 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd nearfull ratio = 0.8 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: osd failsafe full ratio = 0.95 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: osd memory target = 1610612736 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: osd objectstore = bluestore 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr] 2024-09-11T08:38:52.621 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: debug mgr = 20 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: mgr/telemetry/nag = false 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout:[mon] 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: debug ms = 1 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: debug mon = 20 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: debug paxos = 20 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: debug auth = 20 2024-09-11T08:38:52.622 INFO:teuthology.orchestra.run.smithi077.stdout: mon data avail warn = 5 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: mon mgr mkfs grace = 240 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min pgs per osd = 4 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd reporter subtree level = osd 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: mon osd prime pg temp = true 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: mon reweight min bytes per osd = 10 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: # rotate auth tickets quickly to exercise renewal paths 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: auth mon ticket ttl = 660 # 11m 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: auth service ticket ttl = 240 # 4m 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: # don't complain about insecure global_id in the test suite 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim = false 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2024-09-11T08:38:52.623 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: # 1m isn't quite enough 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: mon_down_mkfs_grace = 2m 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: mon_warn_on_filestore_osds = false 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout:[client] 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: rgw cache enabled = true 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable ops log = true 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: rgw enable usage log = true 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2024-09-11T08:38:52.624 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd = 20 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: debug rbd_mirror = 15 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: log to stderr = False 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: rbd default features = 125 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: rbd mirror image policy type = simple 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout:[mon.a] 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.625 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.626 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2024-09-11T08:38:52.626 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.626 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.626 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2024-09-11T08:38:52.626 INFO:teuthology.orchestra.run.smithi077.stdout: admin socket = /var/run/ceph/rbd-mirror.$cluster-$name.asok 2024-09-11T08:38:52.626 INFO:teuthology.orchestra.run.smithi077.stdout: pid file = /var/run/ceph/rbd-mirror.$cluster-$name.pid 2024-09-11T08:38:52.627 INFO:tasks.ceph:Creating admin key on cluster2.mon.a... 2024-09-11T08:38:52.627 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/cluster2.keyring 2024-09-11T08:38:52.718 INFO:tasks.ceph:Copying monmap to all nodes... 2024-09-11T08:38:52.719 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:52.719 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.keyring of=/dev/stdout 2024-09-11T08:38:52.724 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:52.724 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/cluster2.monmap of=/dev/stdout 2024-09-11T08:38:52.771 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T08:38:52.771 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:52.771 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-11T08:38:52.772 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T08:38:52.796 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:38:52.796 DEBUG:teuthology.orchestra.run.smithi073:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:38:52.842 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T08:38:52.843 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:52.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd of=/etc/ceph/cluster2.keyring 2024-09-11T08:38:52.843 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0644 /etc/ceph/cluster2.keyring 2024-09-11T08:38:52.866 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:38:52.867 DEBUG:teuthology.orchestra.run.smithi077:> dd of=/home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:38:52.916 INFO:tasks.ceph:Setting up mon nodes... 2024-09-11T08:38:52.917 INFO:tasks.ceph:Setting up mgr nodes... 2024-09-11T08:38:52.917 DEBUG:teuthology.orchestra.run.smithi073:> sudo mkdir -p /var/lib/ceph/mgr/cluster2-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-11T08:38:52.960 INFO:teuthology.orchestra.run.smithi073.stdout:creating /var/lib/ceph/mgr/cluster2-x/keyring 2024-09-11T08:38:52.963 INFO:tasks.ceph:Setting up mds nodes... 2024-09-11T08:38:52.963 INFO:tasks.ceph_client:Setting up client nodes... 2024-09-11T08:38:52.963 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/cluster2.client.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.0.keyring 2024-09-11T08:38:53.039 INFO:teuthology.orchestra.run.smithi073.stdout:creating /etc/ceph/cluster2.client.0.keyring 2024-09-11T08:38:53.050 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror /etc/ceph/cluster2.client.mirror.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.keyring 2024-09-11T08:38:53.086 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.keyring 2024-09-11T08:38:53.098 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.0 /etc/ceph/cluster2.client.mirror.0.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-11T08:38:53.171 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.0.keyring 2024-09-11T08:38:53.181 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.1 /etc/ceph/cluster2.client.mirror.1.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-11T08:38:53.255 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.1.keyring 2024-09-11T08:38:53.265 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.2 /etc/ceph/cluster2.client.mirror.2.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-11T08:38:53.337 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.2.keyring 2024-09-11T08:38:53.348 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.3 /etc/ceph/cluster2.client.mirror.3.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-11T08:38:53.422 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.3.keyring 2024-09-11T08:38:53.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.4 /etc/ceph/cluster2.client.mirror.4.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-11T08:38:53.507 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.4.keyring 2024-09-11T08:38:53.517 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.5 /etc/ceph/cluster2.client.mirror.5.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-11T08:38:53.590 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.5.keyring 2024-09-11T08:38:53.601 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.mirror.6 /etc/ceph/cluster2.client.mirror.6.keyring && sudo chmod 0644 /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-11T08:38:53.676 INFO:teuthology.orchestra.run.smithi077.stdout:creating /etc/ceph/cluster2.client.mirror.6.keyring 2024-09-11T08:38:53.686 INFO:tasks.ceph:Running mkfs on osd nodes... 2024-09-11T08:38:53.686 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi073.front.sepia.ceph.com'): {'cluster1.osd.0': '/dev/vg_nvme/lv_1', 'cluster1.osd.1': '/dev/vg_nvme/lv_2', 'cluster1.osd.2': '/dev/vg_nvme/lv_3'}, Remote(name='ubuntu@smithi077.front.sepia.ceph.com'): {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'}} 2024-09-11T08:38:53.687 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/cluster2-0 2024-09-11T08:38:53.737 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:53.738 INFO:tasks.ceph:role: cluster2.osd.0 2024-09-11T08:38:53.738 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T08:38:53.738 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T08:38:53.799 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T08:38:54.258 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-09-11T08:38:54.260 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-09-11T08:38:54.260 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/cluster2-0 2024-09-11T08:38:54.406 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-0 2024-09-11T08:38:54.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:38:54.418 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T08:38:54.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/cluster2-1 2024-09-11T08:38:54.470 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:54.470 INFO:tasks.ceph:role: cluster2.osd.1 2024-09-11T08:38:54.470 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T08:38:54.471 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T08:38:54.532 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T08:38:54.533 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T08:38:54.533 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T08:38:54.966 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-09-11T08:38:54.968 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-09-11T08:38:54.968 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/cluster2-1 2024-09-11T08:38:55.036 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-1 2024-09-11T08:38:55.088 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:38:55.088 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T08:38:55.088 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/osd/cluster2-2 2024-09-11T08:38:55.141 INFO:tasks.ceph:roles_to_devs: {'cluster2.osd.0': '/dev/vg_nvme/lv_1', 'cluster2.osd.1': '/dev/vg_nvme/lv_2', 'cluster2.osd.2': '/dev/vg_nvme/lv_3'} 2024-09-11T08:38:55.141 INFO:tasks.ceph:role: cluster2.osd.2 2024-09-11T08:38:55.141 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T08:38:55.141 DEBUG:teuthology.orchestra.run.smithi077:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 attr=2, projid32bit=1 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: = reflink=1 bigtime=0 inobtcount=0 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: = sunit=0 swidth=0 blks 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout:log =internal log bsize=4096 blocks=11443, version=2 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2024-09-11T08:38:55.202 INFO:teuthology.orchestra.run.smithi077.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2024-09-11T08:38:55.651 INFO:teuthology.orchestra.run.smithi077.stdout:Discarding blocks...Done. 2024-09-11T08:38:55.653 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi077.front.sepia.ceph.com -o noatime 2024-09-11T08:38:55.653 DEBUG:teuthology.orchestra.run.smithi077:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/cluster2-2 2024-09-11T08:38:55.747 DEBUG:teuthology.orchestra.run.smithi077:> sudo /sbin/restorecon /var/lib/ceph/osd/cluster2-2 2024-09-11T08:38:55.757 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:38:55.757 INFO:teuthology.orchestra.run.smithi077.stderr:sudo: /sbin/restorecon: command not found 2024-09-11T08:38:55.758 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:38:55.869 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:55.864+0000 7fa4620c37c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-0/keyring: can't open /var/lib/ceph/osd/cluster2-0/keyring: (2) No such file or directory 2024-09-11T08:38:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:55.864+0000 7fa4620c37c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-0/keyring 2024-09-11T08:38:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:55.864+0000 7fa4620c37c0 -1 bdev(0x55df40bc3800 /var/lib/ceph/osd/cluster2-0/block) open stat got: (1) Operation not permitted 2024-09-11T08:38:55.871 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:55.868+0000 7fa4620c37c0 -1 bluestore(/var/lib/ceph/osd/cluster2-0) _read_fsid unparsable uuid 2024-09-11T08:38:59.511 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-0 2024-09-11T08:38:59.524 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:38:59.605 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:59.600+0000 7f67c7e2c7c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-1/keyring: can't open /var/lib/ceph/osd/cluster2-1/keyring: (2) No such file or directory 2024-09-11T08:38:59.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:59.600+0000 7f67c7e2c7c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-1/keyring 2024-09-11T08:38:59.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:59.600+0000 7f67c7e2c7c0 -1 bdev(0x561c83409800 /var/lib/ceph/osd/cluster2-1/block) open stat got: (1) Operation not permitted 2024-09-11T08:38:59.606 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:38:59.600+0000 7f67c7e2c7c0 -1 bluestore(/var/lib/ceph/osd/cluster2-1) _read_fsid unparsable uuid 2024-09-11T08:39:03.047 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-1 2024-09-11T08:39:03.108 DEBUG:teuthology.orchestra.run.smithi077:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster cluster2 --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:39:03.190 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:39:03.184+0000 7f6a494c87c0 -1 auth: error reading file: /var/lib/ceph/osd/cluster2-2/keyring: can't open /var/lib/ceph/osd/cluster2-2/keyring: (2) No such file or directory 2024-09-11T08:39:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:39:03.188+0000 7f6a494c87c0 -1 created new key in keyring /var/lib/ceph/osd/cluster2-2/keyring 2024-09-11T08:39:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:39:03.188+0000 7f6a494c87c0 -1 bdev(0x561f49ac1800 /var/lib/ceph/osd/cluster2-2/block) open stat got: (1) Operation not permitted 2024-09-11T08:39:03.191 INFO:teuthology.orchestra.run.smithi077.stderr:2024-09-11T08:39:03.188+0000 7f6a494c87c0 -1 bluestore(/var/lib/ceph/osd/cluster2-2) _read_fsid unparsable uuid 2024-09-11T08:39:06.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/osd/cluster2-2 2024-09-11T08:39:06.380 INFO:tasks.ceph:Reading keys from all nodes... 2024-09-11T08:39:06.381 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:39:06.381 DEBUG:teuthology.orchestra.run.smithi073:> sudo dd if=/var/lib/ceph/mgr/cluster2-x/keyring of=/dev/stdout 2024-09-11T08:39:06.395 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.395 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster2-0/keyring of=/dev/stdout 2024-09-11T08:39:06.435 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.435 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster2-1/keyring of=/dev/stdout 2024-09-11T08:39:06.491 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.491 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster2-2/keyring of=/dev/stdout 2024-09-11T08:39:06.546 DEBUG:teuthology.orchestra.run.smithi073:> set -ex 2024-09-11T08:39:06.547 DEBUG:teuthology.orchestra.run.smithi073:> dd if=/etc/ceph/cluster2.client.0.keyring of=/dev/stdout 2024-09-11T08:39:06.553 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.553 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.keyring of=/dev/stdout 2024-09-11T08:39:06.595 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.595 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.0.keyring of=/dev/stdout 2024-09-11T08:39:06.643 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.643 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.1.keyring of=/dev/stdout 2024-09-11T08:39:06.691 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.692 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.2.keyring of=/dev/stdout 2024-09-11T08:39:06.739 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.739 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.3.keyring of=/dev/stdout 2024-09-11T08:39:06.787 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.787 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.4.keyring of=/dev/stdout 2024-09-11T08:39:06.835 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.835 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.5.keyring of=/dev/stdout 2024-09-11T08:39:06.883 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:06.883 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/etc/ceph/cluster2.client.mirror.6.keyring of=/dev/stdout 2024-09-11T08:39:06.932 INFO:tasks.ceph:Adding keys to all mons... 2024-09-11T08:39:06.932 DEBUG:teuthology.orchestra.run.smithi077:> sudo tee -a /etc/ceph/cluster2.keyring 2024-09-11T08:39:06.985 INFO:teuthology.orchestra.run.smithi077.stdout:[mgr.x] 2024-09-11T08:39:06.986 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAcV+FmALRXORAAhHyA6rFfRCJU7PHtcM/e8w== 2024-09-11T08:39:06.986 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.0] 2024-09-11T08:39:06.986 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAfV+FmoOTfMxAAE52rF4UMruxnRhHsvH7ipg== 2024-09-11T08:39:06.986 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.1] 2024-09-11T08:39:06.986 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAjV+FmRyUuJBAARC1nWfNWsNPssVpsfw3YHQ== 2024-09-11T08:39:06.986 INFO:teuthology.orchestra.run.smithi077.stdout:[osd.2] 2024-09-11T08:39:06.989 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAnV+FmyWt3CxAA4UOswRnEGxkOlhzzyAj4Cw== 2024-09-11T08:39:06.990 INFO:teuthology.orchestra.run.smithi077.stdout:[client.0] 2024-09-11T08:39:06.990 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmySB2AhAAOXOSceAZQrwFUO80AlV2CQ== 2024-09-11T08:39:06.990 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror] 2024-09-11T08:39:06.990 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmQq5CBRAA26m8e/pfpTRvwkYMSGWwsg== 2024-09-11T08:39:06.990 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2024-09-11T08:39:06.990 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+Fm34xHChAASA95uh893iAW+3s2ltrk3g== 2024-09-11T08:39:06.990 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmES5NDxAAAoW4lvoJ+tDT6RPCcUdclg== 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+Fmm746FBAA79bLJ0CUnWruEWULvoyxNQ== 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+Fmin1HGRAAh7g1Oi3tT5VK8H2X3WlcbQ== 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.4] 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmL35RHhAAlJf7D/ho8UCCU97vuz7urA== 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.5] 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmjhFPIxAAlDJkNJHQd2VAArSursn2YQ== 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.6] 2024-09-11T08:39:06.991 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmcTRiKBAAp2K5agBR4s4Zu/XvoIbnZg== 2024-09-11T08:39:06.992 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2024-09-11T08:39:07.079 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.0 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T08:39:07.166 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.1 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T08:39:07.251 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=osd.2 --cap mon 'allow profile osd' --cap mgr 'allow profile osd' --cap osd 'allow *' 2024-09-11T08:39:07.333 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.416 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.510 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.551 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.636 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.726 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.3 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.813 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.4 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.905 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.5 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:07.949 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/cluster2.keyring --name=client.mirror.6 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2024-09-11T08:39:08.034 INFO:tasks.ceph:Running mkfs on mon nodes... 2024-09-11T08:39:08.035 DEBUG:teuthology.orchestra.run.smithi077:> sudo mkdir -p /var/lib/ceph/mon/cluster2-a 2024-09-11T08:39:08.089 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster cluster2 --mkfs -i a --monmap /home/ubuntu/cephtest/cluster2.monmap --keyring /etc/ceph/cluster2.keyring 2024-09-11T08:39:08.178 DEBUG:teuthology.orchestra.run.smithi077:> sudo chown -R ceph:ceph /var/lib/ceph/mon/cluster2-a 2024-09-11T08:39:08.228 DEBUG:teuthology.orchestra.run.smithi077:> rm -- /home/ubuntu/cephtest/cluster2.monmap 2024-09-11T08:39:08.275 INFO:tasks.ceph:Starting mon daemons in cluster cluster2... 2024-09-11T08:39:08.275 INFO:tasks.ceph.cluster2.mon.a:Restarting daemon 2024-09-11T08:39:08.275 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster cluster2 -i a 2024-09-11T08:39:08.320 INFO:tasks.ceph.cluster2.mon.a:Started 2024-09-11T08:39:08.320 INFO:tasks.ceph:Starting mgr daemons in cluster cluster2... 2024-09-11T08:39:08.320 INFO:tasks.ceph.cluster2.mgr.x:Restarting daemon 2024-09-11T08:39:08.321 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster cluster2 -i x 2024-09-11T08:39:08.323 INFO:tasks.ceph.cluster2.mgr.x:Started 2024-09-11T08:39:08.323 INFO:tasks.ceph:Setting crush tunables to default 2024-09-11T08:39:08.323 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster2 osd crush tunables default 2024-09-11T08:39:08.561 INFO:teuthology.orchestra.run.smithi077.stderr:adjusted tunables profile to default 2024-09-11T08:39:08.578 INFO:tasks.ceph:check_enable_crimson: False 2024-09-11T08:39:08.579 INFO:tasks.ceph:Starting osd daemons in cluster cluster2... 2024-09-11T08:39:08.579 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:08.579 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster2-0/fsid of=/dev/stdout 2024-09-11T08:39:08.590 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:08.590 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster2-1/fsid of=/dev/stdout 2024-09-11T08:39:08.643 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:39:08.643 DEBUG:teuthology.orchestra.run.smithi077:> sudo dd if=/var/lib/ceph/osd/cluster2-2/fsid of=/dev/stdout 2024-09-11T08:39:08.698 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster2 osd new 5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7 0 2024-09-11T08:39:08.737 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:08.736+0000 7f6925385180 -1 mgr[py] Module devicehealth has missing NOTIFY_TYPES member 2024-09-11T08:39:08.947 INFO:teuthology.orchestra.run.smithi077.stdout:0 2024-09-11T08:39:08.964 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster2 osd new f60312d0-3b69-4990-9cb6-df08bf6608b7 1 2024-09-11T08:39:09.135 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.132+0000 7f6925385180 -1 mgr[py] Module iostat has missing NOTIFY_TYPES member 2024-09-11T08:39:09.162 INFO:teuthology.orchestra.run.smithi077.stdout:1 2024-09-11T08:39:09.179 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster2 osd new 9675147d-7ef3-4ef9-bb45-278bb1c44609 2 2024-09-11T08:39:09.276 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.272+0000 7f6925385180 -1 mgr[py] Module telemetry has missing NOTIFY_TYPES member 2024-09-11T08:39:09.383 INFO:teuthology.orchestra.run.smithi077.stdout:2 2024-09-11T08:39:09.401 INFO:tasks.ceph.cluster2.osd.0:Restarting daemon 2024-09-11T08:39:09.401 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 0 2024-09-11T08:39:09.403 INFO:tasks.ceph.cluster2.osd.0:Started 2024-09-11T08:39:09.403 INFO:tasks.ceph.cluster2.osd.1:Restarting daemon 2024-09-11T08:39:09.403 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 1 2024-09-11T08:39:09.405 INFO:tasks.ceph.cluster2.osd.1:Started 2024-09-11T08:39:09.405 INFO:tasks.ceph.cluster2.osd.2:Restarting daemon 2024-09-11T08:39:09.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster cluster2 -i 2 2024-09-11T08:39:09.407 INFO:tasks.ceph.cluster2.osd.2:Started 2024-09-11T08:39:09.407 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T08:39:09.587 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:39:09.588 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"fsid":"600c207a-6d49-48ae-9606-243c4786b265","created":"2024-09-11T08:39:08.452088+0000","modified":"2024-09-11T08:39:09.383976+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T08:39:09.383976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f60312d0-3b69-4990-9cb6-df08bf6608b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9675147d-7ef3-4ef9-bb45-278bb1c44609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:39:09.604 INFO:tasks.ceph.ceph_manager.cluster2:[] 2024-09-11T08:39:09.605 INFO:tasks.ceph:Waiting for OSDs to come up 2024-09-11T08:39:09.693 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.688+0000 7f6925385180 -1 mgr[py] Module osd_support has missing NOTIFY_TYPES member 2024-09-11T08:39:09.743 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.740+0000 7f6925385180 -1 mgr[py] Module selftest has missing NOTIFY_TYPES member 2024-09-11T08:39:09.792 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.788+0000 7f6925385180 -1 mgr[py] Module progress has missing NOTIFY_TYPES member 2024-09-11T08:39:09.865 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.864+0000 7f6925385180 -1 mgr[py] Module crash has missing NOTIFY_TYPES member 2024-09-11T08:39:09.906 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T08:39:09.914 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.912+0000 7f6925385180 -1 mgr[py] Module telegraf has missing NOTIFY_TYPES member 2024-09-11T08:39:09.986 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:09.984+0000 7f6925385180 -1 mgr[py] Module pg_autoscaler has missing NOTIFY_TYPES member 2024-09-11T08:39:10.036 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.032+0000 7f6925385180 -1 mgr[py] Module zabbix has missing NOTIFY_TYPES member 2024-09-11T08:39:10.094 INFO:teuthology.misc.health.smithi077.stdout: 2024-09-11T08:39:10.094 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":5,"fsid":"600c207a-6d49-48ae-9606-243c4786b265","created":"2024-09-11T08:39:08.452088+0000","modified":"2024-09-11T08:39:09.383976+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T08:39:09.383976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":2,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f60312d0-3b69-4990-9cb6-df08bf6608b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9675147d-7ef3-4ef9-bb45-278bb1c44609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:39:10.110 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T08:39:10.173 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.168+0000 7f6925385180 -1 mgr[py] Module nfs has missing NOTIFY_TYPES member 2024-09-11T08:39:10.223 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.220+0000 7f6925385180 -1 mgr[py] Module balancer has missing NOTIFY_TYPES member 2024-09-11T08:39:10.440 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:/usr/lib/python3/dist-packages/scipy/__init__.py:67: UserWarning: NumPy was imported from a Python sub-interpreter but NumPy does not properly support sub-interpreters. This will likely work for most users but might cause hard to track down issues or subtle bugs. A common user of the rare sub-interpreter feature is wsgi which also allows single-interpreter mode. 2024-09-11T08:39:10.440 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:Improvements in the case of bugs are welcome, but is not on the NumPy roadmap, and full support may require significant effort to achieve. 2024-09-11T08:39:10.440 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr: from numpy import show_config as show_numpy_config 2024-09-11T08:39:10.447 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.444+0000 7f6925385180 -1 mgr[py] Module diskprediction_local has missing NOTIFY_TYPES member 2024-09-11T08:39:10.572 INFO:tasks.ceph.cluster2.osd.1.smithi077.stderr:2024-09-11T08:39:10.568+0000 7f35fa4ba7c0 -1 Falling back to public interface 2024-09-11T08:39:10.579 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.576+0000 7f6925385180 -1 mgr[py] Module orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:39:10.688 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.684+0000 7f6925385180 -1 mgr[py] Module rbd_support has missing NOTIFY_TYPES member 2024-09-11T08:39:10.749 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.744+0000 7f6925385180 -1 mgr[py] Module alerts has missing NOTIFY_TYPES member 2024-09-11T08:39:10.880 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:10.876+0000 7f6925385180 -1 mgr[py] Module test_orchestrator has missing NOTIFY_TYPES member 2024-09-11T08:39:11.037 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:39:11.032+0000 7f00e27f17c0 -1 Falling back to public interface 2024-09-11T08:39:11.349 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:11.348+0000 7f6925385180 -1 mgr[py] Module influx has missing NOTIFY_TYPES member 2024-09-11T08:39:11.422 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:11.420+0000 7f6925385180 -1 mgr[py] Module status has missing NOTIFY_TYPES member 2024-09-11T08:39:11.477 INFO:tasks.ceph.cluster2.osd.0.smithi077.stderr:2024-09-11T08:39:11.472+0000 7f2b1527b7c0 -1 Falling back to public interface 2024-09-11T08:39:11.564 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:11.560+0000 7f6925385180 -1 mgr[py] Module volumes has missing NOTIFY_TYPES member 2024-09-11T08:39:11.636 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:11.632+0000 7f6925385180 -1 mgr[py] Module osd_perf_query has missing NOTIFY_TYPES member 2024-09-11T08:39:11.916 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:11.912+0000 7f6925385180 -1 mgr[py] Module prometheus has missing NOTIFY_TYPES member 2024-09-11T08:39:11.969 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:11.968+0000 7f6925385180 -1 mgr[py] Module snap_schedule has missing NOTIFY_TYPES member 2024-09-11T08:39:13.065 INFO:tasks.ceph.cluster2.mgr.x.smithi073.stderr:2024-09-11T08:39:13.064+0000 7f68cbfb8640 -1 mgr.server handle_report got status from non-daemon mon.a 2024-09-11T08:39:15.170 INFO:tasks.ceph.cluster2.osd.1.smithi077.stderr:2024-09-11T08:39:15.164+0000 7f35fa4ba7c0 -1 osd.1 0 log_to_monitors true 2024-09-11T08:39:15.467 INFO:tasks.ceph.cluster2.osd.0.smithi077.stderr:2024-09-11T08:39:15.464+0000 7f2b1527b7c0 -1 osd.0 0 log_to_monitors true 2024-09-11T08:39:15.694 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:39:15.688+0000 7f00e27f17c0 -1 osd.2 0 log_to_monitors true 2024-09-11T08:39:16.414 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T08:39:16.712 INFO:teuthology.misc.health.smithi077.stdout: 2024-09-11T08:39:16.712 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":6,"fsid":"600c207a-6d49-48ae-9606-243c4786b265","created":"2024-09-11T08:39:08.452088+0000","modified":"2024-09-11T08:39:16.032198+0000","last_up_change":"0.000000","last_in_change":"2024-09-11T08:39:09.383976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":0,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[],"osds":[{"osd":0,"uuid":"5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"f60312d0-3b69-4990-9cb6-df08bf6608b7","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"9675147d-7ef3-4ef9-bb45-278bb1c44609","up":0,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:39:16.729 DEBUG:teuthology.misc:0 of 3 OSDs are up 2024-09-11T08:39:18.917 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:39:18.912+0000 7f00defac640 -1 osd.2 0 waiting for initial osdmap 2024-09-11T08:39:18.934 INFO:tasks.ceph.cluster2.osd.1.smithi077.stderr:2024-09-11T08:39:18.928+0000 7f35f5c40640 -1 osd.1 0 waiting for initial osdmap 2024-09-11T08:39:18.947 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:39:18.944+0000 7f00d65be640 -1 osd.2 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:39:18.958 INFO:tasks.ceph.cluster2.osd.0.smithi077.stderr:2024-09-11T08:39:18.952+0000 7f2b11a36640 -1 osd.0 0 waiting for initial osdmap 2024-09-11T08:39:18.963 INFO:tasks.ceph.cluster2.osd.1.smithi077.stderr:2024-09-11T08:39:18.960+0000 7f35ee287640 -1 osd.1 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:39:18.995 INFO:tasks.ceph.cluster2.osd.0.smithi077.stderr:2024-09-11T08:39:18.992+0000 7f2b09048640 -1 osd.0 7 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-09-11T08:39:23.032 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster cluster2 osd dump --format=json 2024-09-11T08:39:23.387 INFO:teuthology.misc.health.smithi077.stdout: 2024-09-11T08:39:23.388 INFO:teuthology.misc.health.smithi077.stdout:{"epoch":11,"fsid":"600c207a-6d49-48ae-9606-243c4786b265","created":"2024-09-11T08:39:08.452088+0000","modified":"2024-09-11T08:39:22.964693+0000","last_up_change":"2024-09-11T08:39:19.960494+0000","last_in_change":"2024-09-11T08:39:09.383976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:39:20.054791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6817","nonce":167757817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6819","nonce":167757817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6823","nonce":167757817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6821","nonce":167757817}]},"public_addr":"172.21.15.77:6817/167757817","cluster_addr":"172.21.15.77:6819/167757817","heartbeat_back_addr":"172.21.15.77:6823/167757817","heartbeat_front_addr":"172.21.15.77:6821/167757817","state":["exists","up"]},{"osd":1,"uuid":"f60312d0-3b69-4990-9cb6-df08bf6608b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":9,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6801","nonce":3684380181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6803","nonce":3684380181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6807","nonce":3684380181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6805","nonce":3684380181}]},"public_addr":"172.21.15.77:6801/3684380181","cluster_addr":"172.21.15.77:6803/3684380181","heartbeat_back_addr":"172.21.15.77:6807/3684380181","heartbeat_front_addr":"172.21.15.77:6805/3684380181","state":["exists","up"]},{"osd":2,"uuid":"9675147d-7ef3-4ef9-bb45-278bb1c44609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6809","nonce":1617494006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6811","nonce":1617494006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6815","nonce":1617494006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6813","nonce":1617494006}]},"public_addr":"172.21.15.77:6809/1617494006","cluster_addr":"172.21.15.77:6811/1617494006","heartbeat_back_addr":"172.21.15.77:6815/1617494006","heartbeat_front_addr":"172.21.15.77:6813/1617494006","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.433046+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.136998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.660776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:39:23.405 DEBUG:teuthology.misc:3 of 3 OSDs are up 2024-09-11T08:39:23.405 INFO:tasks.ceph:Creating RBD pool 2024-09-11T08:39:23.405 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster2 osd pool create rbd 8 2024-09-11T08:39:23.973 INFO:teuthology.orchestra.run.smithi077.stderr:pool 'rbd' created 2024-09-11T08:39:24.000 DEBUG:teuthology.orchestra.run.smithi077:> rbd --cluster cluster2 pool init rbd 2024-09-11T08:39:27.023 INFO:tasks.ceph:Starting mds daemons in cluster cluster2... 2024-09-11T08:39:27.024 INFO:tasks.ceph:Waiting until cluster2 daemons up and pgs clean... 2024-09-11T08:39:27.024 INFO:tasks.ceph.ceph_manager.cluster2:waiting for mgr available 2024-09-11T08:39:27.024 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 mgr dump --format=json 2024-09-11T08:39:27.025 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2024-09-11T08:39:27.371 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:39:27.386 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":5,"flags":0,"active_gid":4101,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6826","nonce":3011727763},{"type":"v1","addr":"172.21.15.73:6827","nonce":3011727763}]},"active_addr":"172.21.15.73:6827/3011727763","active_change":"2024-09-11T08:39:11.973443+0000","active_mgr_features":4540701547738038271,"available":true,"standbys":[],"modules":["iostat","nfs","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2359","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"6","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","read","upmap","upmap-read"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cephadm_log_destination":{"name":"cephadm_log_destination","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":["file","file,syslog","syslog"],"desc":"Destination for cephadm command's persistent logging","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_elasticsearch":{"name":"container_image_elasticsearch","type":"str","level":"advanced","flags":0,"default_value":"quay.io/omrizeneva/elasticsearch:6.8.23","min":"","max":"","enum_allowed":[],"desc":"elasticsearch container image","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/grafana:9.4.12","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_agent":{"name":"container_image_jaeger_agent","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-agent:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger agent container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_collector":{"name":"container_image_jaeger_collector","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-collector:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger collector container image","long_desc":"","tags":[],"see_also":[]},"container_image_jaeger_query":{"name":"container_image_jaeger_query","type":"str","level":"advanced","flags":0,"default_value":"quay.io/jaegertracing/jaeger-query:1.29","min":"","max":"","enum_allowed":[],"desc":"Jaeger query container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.2.4","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_nvmeof":{"name":"container_image_nvmeof","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/nvmeof:1.2.5","min":"","max":"","enum_allowed":[],"desc":"Nvme-of container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:3.0.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_samba":{"name":"container_image_samba","type":"str","level":"advanced","flags":0,"default_value":"quay.io/samba.org/samba-server:devbuilds-centos-amd64","min":"","max":"","enum_allowed":[],"desc":"Samba/SMB container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"grafana_dashboards_path":{"name":"grafana_dashboards_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/grafana/dashboards/ceph-dashboard/","min":"","max":"","enum_allowed":[],"desc":"location of dashboards to include in grafana deployments","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"hw_monitoring":{"name":"hw_monitoring","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Deploy hw monitoring daemon on every host.","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"oob_default_addr":{"name":"oob_default_addr","type":"str","level":"advanced","flags":0,"default_value":"169.254.1.1","min":"","max":"","enum_allowed":[],"desc":"Default address for RedFish API (oob management).","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"secure_monitoring_stack":{"name":"secure_monitoring_stack","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable TLS security for all the monitoring stack daemons","long_desc":"","tags":[],"see_also":[]},"service_discovery_port":{"name":"service_discovery_port","type":"int","level":"advanced","flags":0,"default_value":"8765","min":"","max":"","enum_allowed":[],"desc":"cephadm service discovery port","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_local","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predict_interval":{"name":"predict_interval","type":"str","level":"advanced","flags":0,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"predictor_model":{"name":"predictor_model","type":"str","level":"advanced","flags":0,"default_value":"prophetstor","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"fail_fs":{"name":"fail_fs","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Fail filesystem for rapid multi-rank mds upgrade","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"exclude_perf_counters":{"name":"exclude_perf_counters","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Do not include perf-counters in the metrics output","long_desc":"Gathering perf-counters from a single Prometheus exporter can degrade ceph-mgr performance, especially in large clusters. Instead, Ceph-exporter daemons are now used by default for perf-counter gathering. This should only be disabled when no ceph-exporters are deployed.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard_description":{"name":"leaderboard_description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"periodic_async_work":{"name":"periodic_async_work","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Periodically check for async work","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_no_wait":{"name":"snapshot_clone_no_wait","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Reject subvolume clone request when cloner threads are busy","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"reef":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"squid":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":0,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":1972597019}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":1656401758}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":3770009269}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.73:0","nonce":2391785570}]}]} 2024-09-11T08:39:27.389 INFO:tasks.ceph.ceph_manager.cluster2:mgr available! 2024-09-11T08:39:27.389 INFO:tasks.ceph.ceph_manager.cluster2:waiting for all up 2024-09-11T08:39:27.389 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T08:39:27.661 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:39:27.661 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"fsid":"600c207a-6d49-48ae-9606-243c4786b265","created":"2024-09-11T08:39:08.452088+0000","modified":"2024-09-11T08:39:26.991630+0000","last_up_change":"2024-09-11T08:39:19.960494+0000","last_in_change":"2024-09-11T08:39:09.383976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:39:20.054791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T08:39:23.746149+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6817","nonce":167757817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6819","nonce":167757817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6823","nonce":167757817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6821","nonce":167757817}]},"public_addr":"172.21.15.77:6817/167757817","cluster_addr":"172.21.15.77:6819/167757817","heartbeat_back_addr":"172.21.15.77:6823/167757817","heartbeat_front_addr":"172.21.15.77:6821/167757817","state":["exists","up"]},{"osd":1,"uuid":"f60312d0-3b69-4990-9cb6-df08bf6608b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6801","nonce":3684380181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6803","nonce":3684380181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6807","nonce":3684380181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6805","nonce":3684380181}]},"public_addr":"172.21.15.77:6801/3684380181","cluster_addr":"172.21.15.77:6803/3684380181","heartbeat_back_addr":"172.21.15.77:6807/3684380181","heartbeat_front_addr":"172.21.15.77:6805/3684380181","state":["exists","up"]},{"osd":2,"uuid":"9675147d-7ef3-4ef9-bb45-278bb1c44609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6809","nonce":1617494006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6811","nonce":1617494006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6815","nonce":1617494006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6813","nonce":1617494006}]},"public_addr":"172.21.15.77:6809/1617494006","cluster_addr":"172.21.15.77:6811/1617494006","heartbeat_back_addr":"172.21.15.77:6815/1617494006","heartbeat_front_addr":"172.21.15.77:6813/1617494006","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.433046+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.136998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.660776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:39:27.678 INFO:tasks.ceph.ceph_manager.cluster2:all up! 2024-09-11T08:39:27.678 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T08:39:27.954 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:39:27.954 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":15,"fsid":"600c207a-6d49-48ae-9606-243c4786b265","created":"2024-09-11T08:39:08.452088+0000","modified":"2024-09-11T08:39:26.991630+0000","last_up_change":"2024-09-11T08:39:19.960494+0000","last_in_change":"2024-09-11T08:39:09.383976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:39:20.054791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T08:39:23.746149+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6817","nonce":167757817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6819","nonce":167757817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6823","nonce":167757817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6821","nonce":167757817}]},"public_addr":"172.21.15.77:6817/167757817","cluster_addr":"172.21.15.77:6819/167757817","heartbeat_back_addr":"172.21.15.77:6823/167757817","heartbeat_front_addr":"172.21.15.77:6821/167757817","state":["exists","up"]},{"osd":1,"uuid":"f60312d0-3b69-4990-9cb6-df08bf6608b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6801","nonce":3684380181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6803","nonce":3684380181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6807","nonce":3684380181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6805","nonce":3684380181}]},"public_addr":"172.21.15.77:6801/3684380181","cluster_addr":"172.21.15.77:6803/3684380181","heartbeat_back_addr":"172.21.15.77:6807/3684380181","heartbeat_front_addr":"172.21.15.77:6805/3684380181","state":["exists","up"]},{"osd":2,"uuid":"9675147d-7ef3-4ef9-bb45-278bb1c44609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":12,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6809","nonce":1617494006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6811","nonce":1617494006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6815","nonce":1617494006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6813","nonce":1617494006}]},"public_addr":"172.21.15.77:6809/1617494006","cluster_addr":"172.21.15.77:6811/1617494006","heartbeat_back_addr":"172.21.15.77:6815/1617494006","heartbeat_front_addr":"172.21.15.77:6813/1617494006","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.433046+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.136998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.660776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_removed_snaps":[{"pool":2,"snaps":[{"begin":2,"length":1}]}],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T08:39:27.973 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 flush_pg_stats 2024-09-11T08:39:27.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 flush_pg_stats 2024-09-11T08:39:27.974 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 flush_pg_stats 2024-09-11T08:39:28.126 INFO:teuthology.orchestra.run.smithi077.stdout:34359738373 2024-09-11T08:39:28.126 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T08:39:28.132 INFO:teuthology.orchestra.run.smithi077.stdout:34359738374 2024-09-11T08:39:28.132 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T08:39:28.143 INFO:teuthology.orchestra.run.smithi077.stdout:34359738373 2024-09-11T08:39:28.144 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T08:39:28.405 INFO:teuthology.orchestra.run.smithi077.stdout:34359738372 2024-09-11T08:39:28.422 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-11T08:39:28.463 INFO:teuthology.orchestra.run.smithi077.stdout:34359738372 2024-09-11T08:39:28.469 INFO:teuthology.orchestra.run.smithi077.stdout:34359738373 2024-09-11T08:39:28.482 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-09-11T08:39:28.486 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-09-11T08:39:29.423 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T08:39:29.483 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T08:39:29.487 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T08:39:29.773 INFO:teuthology.orchestra.run.smithi077.stdout:34359738372 2024-09-11T08:39:29.781 INFO:teuthology.orchestra.run.smithi077.stdout:34359738372 2024-09-11T08:39:29.792 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.2 2024-09-11T08:39:29.799 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738372 for osd.0 2024-09-11T08:39:29.847 INFO:teuthology.orchestra.run.smithi077.stdout:34359738373 2024-09-11T08:39:29.863 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738373 for osd.1 2024-09-11T08:39:30.793 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.2 2024-09-11T08:39:30.800 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.0 2024-09-11T08:39:30.863 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd last-stat-seq osd.1 2024-09-11T08:39:31.104 INFO:teuthology.orchestra.run.smithi077.stdout:34359738373 2024-09-11T08:39:31.121 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.2 2024-09-11T08:39:31.122 DEBUG:teuthology.parallel:result is None 2024-09-11T08:39:31.129 INFO:teuthology.orchestra.run.smithi077.stdout:34359738373 2024-09-11T08:39:31.146 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738373 got 34359738373 for osd.0 2024-09-11T08:39:31.146 DEBUG:teuthology.parallel:result is None 2024-09-11T08:39:31.163 INFO:teuthology.orchestra.run.smithi077.stdout:34359738374 2024-09-11T08:39:31.179 INFO:tasks.ceph.ceph_manager.cluster2:need seq 34359738374 got 34359738374 for osd.1 2024-09-11T08:39:31.179 DEBUG:teuthology.parallel:result is None 2024-09-11T08:39:31.180 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-11T08:39:31.180 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T08:39:31.517 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:39:31.518 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-09-11T08:39:31.534 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T08:39:29.985389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82640,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032880,"statfs":{"total":289910292480,"available":289825669120,"internally_reserved":0,"allocated":1581056,"data_stored":1292397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":82963792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.014215+0000","last_change":"2024-09-11T08:39:27.014675+0000","last_active":"2024-09-11T08:39:27.014215+0000","last_peered":"2024-09-11T08:39:27.014215+0000","last_clean":"2024-09-11T08:39:27.014215+0000","last_became_active":"2024-09-11T08:39:24.993902+0000","last_became_peered":"2024-09-11T08:39:24.993902+0000","last_unstale":"2024-09-11T08:39:27.014215+0000","last_undegraded":"2024-09-11T08:39:27.014215+0000","last_fullsized":"2024-09-11T08:39:27.014215+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:22:21.610509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.014846+0000","last_change":"2024-09-11T08:39:27.015272+0000","last_active":"2024-09-11T08:39:27.014846+0000","last_peered":"2024-09-11T08:39:27.014846+0000","last_clean":"2024-09-11T08:39:27.014846+0000","last_became_active":"2024-09-11T08:39:24.995216+0000","last_became_peered":"2024-09-11T08:39:24.995216+0000","last_unstale":"2024-09-11T08:39:27.014846+0000","last_undegraded":"2024-09-11T08:39:27.014846+0000","last_fullsized":"2024-09-11T08:39:27.014846+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:45:49.872594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.013726+0000","last_change":"2024-09-11T08:39:27.014087+0000","last_active":"2024-09-11T08:39:27.013726+0000","last_peered":"2024-09-11T08:39:27.013726+0000","last_clean":"2024-09-11T08:39:27.013726+0000","last_became_active":"2024-09-11T08:39:24.994002+0000","last_became_peered":"2024-09-11T08:39:24.994002+0000","last_unstale":"2024-09-11T08:39:27.013726+0000","last_undegraded":"2024-09-11T08:39:27.013726+0000","last_fullsized":"2024-09-11T08:39:27.013726+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:54:18.729378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070708900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.013567+0000","last_change":"2024-09-11T08:39:27.014269+0000","last_active":"2024-09-11T08:39:27.013567+0000","last_peered":"2024-09-11T08:39:27.013567+0000","last_clean":"2024-09-11T08:39:27.013567+0000","last_became_active":"2024-09-11T08:39:24.995202+0000","last_became_peered":"2024-09-11T08:39:24.995202+0000","last_unstale":"2024-09-11T08:39:27.013567+0000","last_undegraded":"2024-09-11T08:39:27.013567+0000","last_fullsized":"2024-09-11T08:39:27.013567+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:38.241377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010507380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.008400+0000","last_change":"2024-09-11T08:39:27.008400+0000","last_active":"2024-09-11T08:39:27.008400+0000","last_peered":"2024-09-11T08:39:27.008400+0000","last_clean":"2024-09-11T08:39:27.008400+0000","last_became_active":"2024-09-11T08:39:24.990422+0000","last_became_peered":"2024-09-11T08:39:24.990422+0000","last_unstale":"2024-09-11T08:39:27.008400+0000","last_undegraded":"2024-09-11T08:39:27.008400+0000","last_fullsized":"2024-09-11T08:39:27.008400+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:59:33.107916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000934803,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.657448+0000","last_change":"2024-09-11T08:39:27.657840+0000","last_active":"2024-09-11T08:39:27.657448+0000","last_peered":"2024-09-11T08:39:27.657448+0000","last_clean":"2024-09-11T08:39:27.657448+0000","last_became_active":"2024-09-11T08:39:24.994237+0000","last_became_peered":"2024-09-11T08:39:24.994237+0000","last_unstale":"2024-09-11T08:39:27.657448+0000","last_undegraded":"2024-09-11T08:39:27.657448+0000","last_fullsized":"2024-09-11T08:39:27.657448+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:53.237878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063005199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.657372+0000","last_change":"2024-09-11T08:39:27.657633+0000","last_active":"2024-09-11T08:39:27.657372+0000","last_peered":"2024-09-11T08:39:27.657372+0000","last_clean":"2024-09-11T08:39:27.657372+0000","last_became_active":"2024-09-11T08:39:24.990604+0000","last_became_peered":"2024-09-11T08:39:24.990604+0000","last_unstale":"2024-09-11T08:39:27.657372+0000","last_undegraded":"2024-09-11T08:39:27.657372+0000","last_fullsized":"2024-09-11T08:39:27.657372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:38.112533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000543883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.014416+0000","last_change":"2024-09-11T08:39:27.015047+0000","last_active":"2024-09-11T08:39:27.014416+0000","last_peered":"2024-09-11T08:39:27.014416+0000","last_clean":"2024-09-11T08:39:27.014416+0000","last_became_active":"2024-09-11T08:39:24.987701+0000","last_became_peered":"2024-09-11T08:39:24.987701+0000","last_unstale":"2024-09-11T08:39:27.014416+0000","last_undegraded":"2024-09-11T08:39:27.014416+0000","last_fullsized":"2024-09-11T08:39:27.014416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:55:50.605878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016047609999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":146,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.012945+0000","last_change":"2024-09-11T08:39:22.547555+0000","last_active":"2024-09-11T08:39:27.012945+0000","last_peered":"2024-09-11T08:39:27.012945+0000","last_clean":"2024-09-11T08:39:27.012945+0000","last_became_active":"2024-09-11T08:39:22.544974+0000","last_became_peered":"2024-09-11T08:39:22.544974+0000","last_unstale":"2024-09-11T08:39:27.012945+0000","last_undegraded":"2024-09-11T08:39:27.012945+0000","last_fullsized":"2024-09-11T08:39:27.012945+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_clean_scrub_stamp":"2024-09-11T08:39:20.961806+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:01.190641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:39:31.535 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T08:39:31.800 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:39:31.801 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-09-11T08:39:31.817 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":18,"stamp":"2024-09-11T08:39:29.985389+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":82640,"kb_used_data":1544,"kb_used_omap":4,"kb_used_meta":81019,"kb_avail":283032880,"statfs":{"total":289910292480,"available":289825669120,"internally_reserved":0,"allocated":1581056,"data_stored":1292397,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4784,"internal_metadata":82963792},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":4,"apply_latency_ms":4,"commit_latency_ns":4000000,"apply_latency_ns":4000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001047"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.014215+0000","last_change":"2024-09-11T08:39:27.014675+0000","last_active":"2024-09-11T08:39:27.014215+0000","last_peered":"2024-09-11T08:39:27.014215+0000","last_clean":"2024-09-11T08:39:27.014215+0000","last_became_active":"2024-09-11T08:39:24.993902+0000","last_became_peered":"2024-09-11T08:39:24.993902+0000","last_unstale":"2024-09-11T08:39:27.014215+0000","last_undegraded":"2024-09-11T08:39:27.014215+0000","last_fullsized":"2024-09-11T08:39:27.014215+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:22:21.610509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.6","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.014846+0000","last_change":"2024-09-11T08:39:27.015272+0000","last_active":"2024-09-11T08:39:27.014846+0000","last_peered":"2024-09-11T08:39:27.014846+0000","last_clean":"2024-09-11T08:39:27.014846+0000","last_became_active":"2024-09-11T08:39:24.995216+0000","last_became_peered":"2024-09-11T08:39:24.995216+0000","last_unstale":"2024-09-11T08:39:27.014846+0000","last_undegraded":"2024-09-11T08:39:27.014846+0000","last_fullsized":"2024-09-11T08:39:27.014846+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:45:49.872594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.5","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.013726+0000","last_change":"2024-09-11T08:39:27.014087+0000","last_active":"2024-09-11T08:39:27.013726+0000","last_peered":"2024-09-11T08:39:27.013726+0000","last_clean":"2024-09-11T08:39:27.013726+0000","last_became_active":"2024-09-11T08:39:24.994002+0000","last_became_peered":"2024-09-11T08:39:24.994002+0000","last_unstale":"2024-09-11T08:39:27.013726+0000","last_undegraded":"2024-09-11T08:39:27.013726+0000","last_fullsized":"2024-09-11T08:39:27.013726+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:54:18.729378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070708900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.4","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.013567+0000","last_change":"2024-09-11T08:39:27.014269+0000","last_active":"2024-09-11T08:39:27.013567+0000","last_peered":"2024-09-11T08:39:27.013567+0000","last_clean":"2024-09-11T08:39:27.013567+0000","last_became_active":"2024-09-11T08:39:24.995202+0000","last_became_peered":"2024-09-11T08:39:24.995202+0000","last_unstale":"2024-09-11T08:39:27.013567+0000","last_undegraded":"2024-09-11T08:39:27.013567+0000","last_fullsized":"2024-09-11T08:39:27.013567+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:38.241377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010507380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.2","version":"15'2","reported_seq":21,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.008400+0000","last_change":"2024-09-11T08:39:27.008400+0000","last_active":"2024-09-11T08:39:27.008400+0000","last_peered":"2024-09-11T08:39:27.008400+0000","last_clean":"2024-09-11T08:39:27.008400+0000","last_became_active":"2024-09-11T08:39:24.990422+0000","last_became_peered":"2024-09-11T08:39:24.990422+0000","last_unstale":"2024-09-11T08:39:27.008400+0000","last_undegraded":"2024-09-11T08:39:27.008400+0000","last_fullsized":"2024-09-11T08:39:27.008400+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:59:33.107916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000934803,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.1","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.657448+0000","last_change":"2024-09-11T08:39:27.657840+0000","last_active":"2024-09-11T08:39:27.657448+0000","last_peered":"2024-09-11T08:39:27.657448+0000","last_clean":"2024-09-11T08:39:27.657448+0000","last_became_active":"2024-09-11T08:39:24.994237+0000","last_became_peered":"2024-09-11T08:39:24.994237+0000","last_unstale":"2024-09-11T08:39:27.657448+0000","last_undegraded":"2024-09-11T08:39:27.657448+0000","last_fullsized":"2024-09-11T08:39:27.657448+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:53.237878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063005199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.0","version":"0'0","reported_seq":19,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.657372+0000","last_change":"2024-09-11T08:39:27.657633+0000","last_active":"2024-09-11T08:39:27.657372+0000","last_peered":"2024-09-11T08:39:27.657372+0000","last_clean":"2024-09-11T08:39:27.657372+0000","last_became_active":"2024-09-11T08:39:24.990604+0000","last_became_peered":"2024-09-11T08:39:24.990604+0000","last_unstale":"2024-09-11T08:39:27.657372+0000","last_undegraded":"2024-09-11T08:39:27.657372+0000","last_fullsized":"2024-09-11T08:39:27.657372+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:38.112533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000543883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"2.3","version":"13'1","reported_seq":20,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.014416+0000","last_change":"2024-09-11T08:39:27.015047+0000","last_active":"2024-09-11T08:39:27.014416+0000","last_peered":"2024-09-11T08:39:27.014416+0000","last_clean":"2024-09-11T08:39:27.014416+0000","last_became_active":"2024-09-11T08:39:24.987701+0000","last_became_peered":"2024-09-11T08:39:24.987701+0000","last_unstale":"2024-09-11T08:39:27.014416+0000","last_undegraded":"2024-09-11T08:39:27.014416+0000","last_fullsized":"2024-09-11T08:39:27.014416+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:55:50.605878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016047609999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[{"start":"2","length":"1"}]},{"pgid":"1.0","version":"10'97","reported_seq":146,"reported_epoch":15,"state":"active+clean","last_fresh":"2024-09-11T08:39:27.012945+0000","last_change":"2024-09-11T08:39:22.547555+0000","last_active":"2024-09-11T08:39:27.012945+0000","last_peered":"2024-09-11T08:39:27.012945+0000","last_clean":"2024-09-11T08:39:27.012945+0000","last_became_active":"2024-09-11T08:39:22.544974+0000","last_became_peered":"2024-09-11T08:39:22.544974+0000","last_unstale":"2024-09-11T08:39:27.012945+0000","last_undegraded":"2024-09-11T08:39:27.012945+0000","last_fullsized":"2024-09-11T08:39:27.012945+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_clean_scrub_stamp":"2024-09-11T08:39:20.961806+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:01.190641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738373,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27148,"kb_used_data":120,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344692,"statfs":{"total":96636764160,"available":96608964608,"internally_reserved":0,"allocated":122880,"data_stored":34833,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1594,"internal_metadata":27654598},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738374,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27752,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344088,"statfs":{"total":96636764160,"available":96608346112,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738373,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":27740,"kb_used_data":712,"kb_used_omap":1,"kb_used_meta":27006,"kb_avail":94344100,"statfs":{"total":96636764160,"available":96608358400,"internally_reserved":0,"allocated":729088,"data_stored":628782,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":27654597},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":2,"apply_latency_ms":2,"commit_latency_ns":2000000,"apply_latency_ns":2000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T08:39:31.818 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-11T08:39:31.818 INFO:tasks.ceph:Waiting until ceph cluster cluster2 is healthy... 2024-09-11T08:39:31.818 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy 2024-09-11T08:39:31.818 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 health --format=json 2024-09-11T08:39:32.126 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T08:39:32.126 INFO:teuthology.orchestra.run.smithi077.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2024-09-11T08:39:32.143 INFO:tasks.ceph.ceph_manager.cluster2:wait_until_healthy done 2024-09-11T08:39:32.144 INFO:teuthology.run_tasks:Running task exec... 2024-09-11T08:39:32.153 INFO:teuthology.task.exec:Executing custom commands... 2024-09-11T08:39:32.153 INFO:teuthology.task.exec:Running commands on role cluster1.client.mirror.0 host ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T08:39:32.153 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:32.499 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror] 2024-09-11T08:39:32.500 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD0VuFmBQD7NBAAoj9BW40sjpRWncf16yvreA== 2024-09-11T08:39:32.500 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-11T08:39:32.500 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:32.500 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror 2024-09-11T08:39:32.518 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:32.875 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2024-09-11T08:39:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD1VuFmjU27ABAA3/f5R/9O5Xkf0dDZ28Efwg== 2024-09-11T08:39:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:32.875 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:32.876 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.0 2024-09-11T08:39:32.893 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:33.238 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2024-09-11T08:39:33.239 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD1VuFmJQe+BRAA9YAzwwmEdvXnwEGRXvMYSg== 2024-09-11T08:39:33.239 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:33.239 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:33.239 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.1 2024-09-11T08:39:33.256 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:33.611 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2024-09-11T08:39:33.611 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD1VuFms6qbChAAtYuBUZGXSvUb+867m857VQ== 2024-09-11T08:39:33.611 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:33.612 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:33.612 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.2 2024-09-11T08:39:33.629 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster1 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:33.971 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2024-09-11T08:39:33.971 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQD1VuFmgBdKDxAAyYjRo1IlxwoKZ33G8zQ92g== 2024-09-11T08:39:33.971 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:33.971 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:33.971 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.3 2024-09-11T08:39:33.989 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror mon '"'"'profile rbd-mirror-peer'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:34.352 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror] 2024-09-11T08:39:34.352 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmQq5CBRAA26m8e/pfpTRvwkYMSGWwsg== 2024-09-11T08:39:34.352 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror-peer" 2024-09-11T08:39:34.352 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:34.352 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror 2024-09-11T08:39:34.372 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.0 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:34.738 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.0] 2024-09-11T08:39:34.739 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+Fm34xHChAASA95uh893iAW+3s2ltrk3g== 2024-09-11T08:39:34.739 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:34.739 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:34.739 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.0 2024-09-11T08:39:34.757 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.1 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:35.118 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.1] 2024-09-11T08:39:35.118 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+FmES5NDxAAAoW4lvoJ+tDT6RPCcUdclg== 2024-09-11T08:39:35.118 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:35.118 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:35.119 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.1 2024-09-11T08:39:35.136 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.2 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:35.476 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.2] 2024-09-11T08:39:35.476 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+Fmm746FBAA79bLJ0CUnWruEWULvoyxNQ== 2024-09-11T08:39:35.476 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:35.476 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:35.476 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.2 2024-09-11T08:39:35.495 DEBUG:teuthology.orchestra.run.smithi077:> sudo TESTDIR=/home/ubuntu/cephtest bash -c 'sudo ceph --cluster cluster2 auth caps client.mirror.3 mon '"'"'profile rbd-mirror'"'"' osd '"'"'profile rbd'"'"'' 2024-09-11T08:39:35.856 INFO:teuthology.orchestra.run.smithi077.stdout:[client.mirror.3] 2024-09-11T08:39:35.856 INFO:teuthology.orchestra.run.smithi077.stdout: key = AQAdV+Fmin1HGRAAh7g1Oi3tT5VK8H2X3WlcbQ== 2024-09-11T08:39:35.856 INFO:teuthology.orchestra.run.smithi077.stdout: caps mon = "profile rbd-mirror" 2024-09-11T08:39:35.856 INFO:teuthology.orchestra.run.smithi077.stdout: caps osd = "profile rbd" 2024-09-11T08:39:35.856 INFO:teuthology.orchestra.run.smithi077.stderr:updated caps for client.mirror.3 2024-09-11T08:39:35.874 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:35.889 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:39:35.890 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:39:35.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:39:35.892 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:35.936 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-11T08:39:35.936 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-11T08:39:35.939 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-11T08:39:35.939 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:35.963 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T08:39:35.964 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T08:39:35.966 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T08:39:35.966 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:35.986 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-11T08:39:35.986 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-11T08:39:35.988 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-11T08:39:35.988 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:36.016 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T08:39:36.016 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T08:39:36.018 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T08:39:36.018 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:36.038 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T08:39:36.038 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T08:39:36.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T08:39:36.040 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:36.059 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-11T08:39:36.059 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-11T08:39:36.061 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-11T08:39:36.062 INFO:teuthology.run_tasks:Running task rbd-mirror... 2024-09-11T08:39:36.071 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T08:39:36.071 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T08:39:36.073 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T08:39:36.073 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-11T08:39:36.115 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726043976 2024-09-11T08:39:36.116 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-11T08:39:36.116 INFO:teuthology.run_tasks:Running task rbd-mirror-thrash... 2024-09-11T08:39:36.168 INFO:tasks.rbd_mirror_thrash:rbd_mirror_thrash using random seed: 1726043976 2024-09-11T08:39:36.168 DEBUG:tasks.rbd_mirror_thrash:Yielding 2024-09-11T08:39:36.169 INFO:teuthology.run_tasks:Running task workunit... 2024-09-11T08:39:36.208 INFO:tasks.workunit:Pulling workunits from ref daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T08:39:36.208 INFO:tasks.workunit:Making a separate scratch dir for every client... 2024-09-11T08:39:36.209 DEBUG:teuthology.orchestra.run.smithi077:> stat -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T08:39:36.210 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster1 2024-09-11T08:39:36.210 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-11T08:39:36.210 INFO:tasks.rbd_mirror_thrash:starting thrash for cluster cluster2 2024-09-11T08:39:36.210 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-11T08:39:36.215 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T08:39:36.216 INFO:teuthology.orchestra.run.smithi077.stderr:stat: cannot statx '/home/ubuntu/cephtest/mnt.cluster1.mirror': No such file or directory 2024-09-11T08:39:36.216 DEBUG:teuthology.orchestra.run.smithi077:> mkdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T08:39:36.263 INFO:tasks.workunit:Created dir /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T08:39:36.263 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror && mkdir -- client.mirror 2024-09-11T08:39:36.311 INFO:tasks.workunit:timeout=6h 2024-09-11T08:39:36.311 INFO:tasks.workunit:cleanup=True 2024-09-11T08:39:36.311 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf /home/ubuntu/cephtest/clone.cluster1.client.mirror && git clone https://git.ceph.com/ceph.git /home/ubuntu/cephtest/clone.cluster1.client.mirror && cd /home/ubuntu/cephtest/clone.cluster1.client.mirror && git checkout daaf78de334647fc0c800f4ac495cc9475a26bae 2024-09-11T08:39:36.365 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Cloning into '/home/ubuntu/cephtest/clone.cluster1.client.mirror'... 2024-09-11T08:40:30.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Note: switching to 'daaf78de334647fc0c800f4ac495cc9475a26bae'. 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:You are in 'detached HEAD' state. You can look around, make experimental 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:changes and commit them, and you can discard any commits you make in this 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:state without impacting any branches by switching back to a branch. 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:If you want to create a new branch to retain commits you create, you may 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:do so (now or later) by using -c with the switch command. Example: 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: git switch -c 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:40:30.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Or undo this operation with: 2024-09-11T08:40:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:40:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: git switch - 2024-09-11T08:40:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:40:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Turn off this advice by setting config variable advice.detachedHead to false 2024-09-11T08:40:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:40:30.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:HEAD is now at daaf78de334 Merge pull request #59682 from zdover23/wip-doc-2024-09-10-backport-59677-to-squid 2024-09-11T08:40:30.367 DEBUG:teuthology.orchestra.run.smithi077:> cd -- /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits && if test -e Makefile ; then make ; fi && find -executable -type f -printf '%P\0' >/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror 2024-09-11T08:40:30.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:for d in direct_io fs ; do ( cd $d ; make all ) ; done 2024-09-11T08:40:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-11T08:40:30.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE direct_io_test.c -o direct_io_test 2024-09-11T08:40:31.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_sync_io.c -o test_sync_io 2024-09-11T08:40:31.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_short_dio_read.c -o test_short_dio_read 2024-09-11T08:40:31.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/direct_io' 2024-09-11T08:40:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:make[1]: Entering directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-11T08:40:31.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:cc -Wall -Wextra -D_GNU_SOURCE test_o_trunc.c -o test_o_trunc 2024-09-11T08:40:31.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:make[1]: Leaving directory '/home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/fs' 2024-09-11T08:40:31.358 DEBUG:teuthology.orchestra.run.smithi077:> set -ex 2024-09-11T08:40:31.358 DEBUG:teuthology.orchestra.run.smithi077:> dd if=/home/ubuntu/cephtest/workunits.list.cluster1.client.mirror of=/dev/stdout 2024-09-11T08:40:31.408 INFO:tasks.workunit:Running workunits matching rbd/rbd_mirror_stress.sh on cluster1.client.mirror... 2024-09-11T08:40:31.409 INFO:tasks.workunit:Running workunit rbd/rbd_mirror_stress.sh... 2024-09-11T08:40:31.409 DEBUG:teuthology.orchestra.run.smithi077:workunit test rbd/rbd_mirror_stress.sh> mkdir -p -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && cd -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp && CEPH_CLI_TEST_DUP_COMMAND=1 CEPH_REF=daaf78de334647fc0c800f4ac495cc9475a26bae TESTDIR="/home/ubuntu/cephtest" CEPH_ARGS="--cluster cluster1" CEPH_ID="mirror" PATH=$PATH:/usr/sbin CEPH_BASE=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror CEPH_MNT=/home/ubuntu/cephtest/mnt.cluster1.mirror CEPH_ARGS='' MIRROR_POOL_MODE=image RBD_IMAGE_FEATURES=layering,exclusive-lock RBD_MIRROR_INSTANCES=4 RBD_MIRROR_MODE=snapshot RBD_MIRROR_USE_EXISTING_CLUSTER=1 RBD_MIRROR_USE_RBD_MIRROR=1 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 6h /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T08:40:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ IMAGE_COUNT=50 2024-09-11T08:40:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ export LOCKDEP=0 2024-09-11T08:40:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ LOCKDEP=0 2024-09-11T08:40:31.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T08:40:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ . /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_helpers.sh 2024-09-11T08:40:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ type xmlstarlet 2024-09-11T08:40:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ XMLSTARLET=xmlstarlet 2024-09-11T08:40:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ RBD_MIRROR_INSTANCES=4 2024-09-11T08:40:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CLUSTER1=cluster1 2024-09-11T08:40:31.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CLUSTER2=cluster2 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ PEER_CLUSTER_SUFFIX= 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ POOL=mirror 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ PARENT_POOL=mirror_parent 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ NS1=ns1 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ NS2=ns2 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ TEMPDIR= 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ID=mirror 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ RBD_IMAGE_FEATURES=layering,exclusive-lock 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ MIRROR_USER_ID_PREFIX=mirror. 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ RBD_MIRROR_MODE=snapshot 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ MIRROR_POOL_MODE=image 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' snapshot = snapshot ']' 2024-09-11T08:40:31.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ MIRROR_POOL_MODE=image 2024-09-11T08:40:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ export 'CEPH_ARGS=--id mirror' 2024-09-11T08:40:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS='--id mirror' 2024-09-11T08:40:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ LAST_MIRROR_INSTANCE=3 2024-09-11T08:40:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ dirname /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/rbd_mirror_stress.sh 2024-09-11T08:40:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ readlink -f /home/ubuntu/cephtest/clone.cluster1.client.mirror/qa/workunits/rbd/../../../src 2024-09-11T08:40:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ROOT=/home/ubuntu/cephtest/clone.cluster1.client.mirror/src 2024-09-11T08:40:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_BIN=. 2024-09-11T08:40:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_SRC=. 2024-09-11T08:40:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' -e CMakeCache.txt ']' 2024-09-11T08:40:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 -gt 0 ']' 2024-09-11T08:40:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ setup 2024-09-11T08:40:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local c 2024-09-11T08:40:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ trap 'cleanup $?' INT TERM EXIT 2024-09-11T08:40:31.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ setup_tempdir 2024-09-11T08:40:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mktemp -d 2024-09-11T08:40:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ TEMPDIR=/tmp/tmp.LFCa6yxq8J 2024-09-11T08:40:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -z 1 ']' 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ setup_pools cluster1 cluster2 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local remote_cluster=cluster2 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mon_map_file 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mon_addr 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local admin_key_file 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local uuid 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster1 osd pool create mirror 64 64 2024-09-11T08:40:32.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror' already exists 2024-09-11T08:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:32.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster1 osd pool create mirror_parent 64 64 2024-09-11T08:40:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror_parent' already exists 2024-09-11T08:40:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:33.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 pool init mirror 2024-09-11T08:40:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:36.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 pool init mirror_parent 2024-09-11T08:40:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:39.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 mirror pool enable --site-name cluster1 mirror image 2024-09-11T08:40:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 mirror pool enable mirror_parent image 2024-09-11T08:40:39.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 namespace create mirror/ns1 2024-09-11T08:40:39.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 namespace create mirror/ns2 2024-09-11T08:40:40.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns1 image 2024-09-11T08:40:40.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 mirror pool enable mirror/ns2 image 2024-09-11T08:40:40.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -z ']' 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -z ']' 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_add cluster1 mirror cluster2 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local client_cluster=cluster2 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local remote_cluster=cluster2 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local uuid_var_name 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local error_code 2024-09-11T08:40:40.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local peer_uuid 2024-09-11T08:40:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T08:40:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:40:40.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror cluster2 2024-09-11T08:40:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_uuid=a021640d-daed-492a-83a4-e9ae9de14059 2024-09-11T08:40:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T08:40:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:40:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T08:40:40.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 -eq 0 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_add cluster1 mirror_parent cluster2 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror_parent 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local client_cluster=cluster2 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local remote_cluster=cluster2 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local uuid_var_name 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:40.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local error_code 2024-09-11T08:40:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local peer_uuid 2024-09-11T08:40:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T08:40:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:40:40.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool peer add mirror_parent cluster2 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_uuid=31837f75-1028-4b2d-998b-8d2d0a2f5c38 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 -eq 0 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:40:40.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ setup_pools cluster2 cluster1 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local remote_cluster=cluster1 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mon_map_file 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mon_addr 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local admin_key_file 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local uuid 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:40.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster2 osd pool create mirror 64 64 2024-09-11T08:40:41.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror' already exists 2024-09-11T08:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:41.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster2 osd pool create mirror_parent 64 64 2024-09-11T08:40:42.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror_parent' already exists 2024-09-11T08:40:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:42.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 pool init mirror 2024-09-11T08:40:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:45.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 pool init mirror_parent 2024-09-11T08:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T08:40:48.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror pool enable --site-name cluster2 mirror image 2024-09-11T08:40:48.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror pool enable mirror_parent image 2024-09-11T08:40:48.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 namespace create mirror/ns1 2024-09-11T08:40:48.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 namespace create mirror/ns2 2024-09-11T08:40:48.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns1 image 2024-09-11T08:40:48.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror pool enable mirror/ns2 image 2024-09-11T08:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -z ']' 2024-09-11T08:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -z ']' 2024-09-11T08:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_add cluster2 mirror cluster1 2024-09-11T08:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:40:48.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local client_cluster=cluster1 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local remote_cluster=cluster1 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local uuid_var_name 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local error_code 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local peer_uuid 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:40:48.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror cluster1 2024-09-11T08:40:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_uuid=577ec95d-e9a7-4d0c-9736-c507ff5d5938 2024-09-11T08:40:48.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 -eq 0 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_add cluster2 mirror_parent cluster1 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:40:48.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror_parent 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local client_cluster=cluster1 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local remote_cluster=cluster1 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local uuid_var_name 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local error_code 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local peer_uuid 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 16 32 2024-09-11T08:40:48.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:40:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster2 mirror pool peer add mirror_parent cluster1 2024-09-11T08:40:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ peer_uuid=a79ea33d-4f3a-4106-9a1e-c3ab32f5c2fa 2024-09-11T08:40:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T08:40:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:40:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 17 ']' 2024-09-11T08:40:48.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 -eq 0 2024-09-11T08:40:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:40:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:40:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirrors cluster1 2024-09-11T08:40:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:40:48.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 0 3 2024-09-11T08:40:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster1:0 2024-09-11T08:40:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:0 2024-09-11T08:40:48.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster1:0 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster1 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=0 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster1 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=0 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster1:1 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:1 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster1:1 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster1 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=1 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:48.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster1 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=1 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster1:2 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:2 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster1:2 cluster instance 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster1:2 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster1 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=2 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster1 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=2 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster1:3 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:3 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster1:3 cluster instance 2024-09-11T08:40:48.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster1:3 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster1 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=3 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster1 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=3 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirrors cluster2 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:40:48.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 0 3 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster2:0 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:0 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster2:0 cluster instance 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster2:0 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster2 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=0 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 = cluster2:0 ']' 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster2 2024-09-11T08:40:48.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster2 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=0 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster2:1 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:1 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster2:1 cluster instance 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster2:1 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster2 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=1 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 1 = cluster2:1 ']' 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster2 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster2 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=1 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster2:2 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:2 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster2:2 cluster instance 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster2:2 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster2 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=2 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 2 = cluster2:2 ']' 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster2 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster2 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=2 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:48.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster2:3 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:3 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster2:3 cluster instance 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster2:3 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster2 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=3 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 3 = cluster2:3 ']' 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster2 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster2 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=3 2024-09-11T08:40:48.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T08:40:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T08:40:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T08:40:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ testlog 'TEST: add image and test replay after client crashes' 2024-09-11T08:40:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ tee -a /tmp/tmp.LFCa6yxq8J/rbd-mirror.test.log 2024-09-11T08:40:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ date '+%F %T' 2024-09-11T08:40:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo 2024-09-11 08:40:48 TEST: add image and test replay after client crashes 2024-09-11T08:40:48.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:2024-09-11 08:40:48 TEST: add image and test replay after client crashes 2024-09-11T08:40:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=test 2024-09-11T08:40:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror test snapshot 512M 2024-09-11T08:40:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:40:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror test 512M 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:40:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:40:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T08:40:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T08:40:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:40:48.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/test 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror test snapshot 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:40:49.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/test snapshot 2024-09-11T08:40:49.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:40:49.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/test 2024-09-11T08:40:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:40:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:40:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:40:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:40:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ac90a74833 2024-09-11T08:40:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ac90a74833 2024-09-11T08:40:49.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:40:49 2024 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:40:49 2024 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:40:49 2024 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:40:49.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:40:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:40:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:40:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:40:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:40:50.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:40:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:40:50.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:40:50.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:40:50.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:40:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:40:50.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:40:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:50.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:50.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:40:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:50.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:40:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:40:50.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:40:50.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:50.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:40:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:40:50.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:40:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:50.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:40:50.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T08:40:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T08:40:50.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T08:40:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T08:40:50.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T08:40:50.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T08:40:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:50.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T08:40:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:51.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:51.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T08:40:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T08:40:51.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T08:40:51.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:51.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T08:40:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T08:40:51.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T08:40:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:51.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T08:40:51.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:51.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:40:51.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 1 2024-09-11T08:40:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:40:52.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:40:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:52.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:40:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:40:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:40:52.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:52.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:40:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:40:52.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:40:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:52.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:40:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:40:52.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:52.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:40:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:40:52.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:40:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:52.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:52.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:40:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:52.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:52.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:52.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T08:40:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T08:40:52.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T08:40:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:52.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T08:40:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:52.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:52.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T08:40:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T08:40:52.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T08:40:52.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:52.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T08:40:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T08:40:52.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T08:40:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:52.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T08:40:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:52.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:40:52.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 2 2024-09-11T08:40:54.211 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3165567076388388) > weight (0.25) 2024-09-11T08:40:54.211 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8805499029792928) > weight (0.25) 2024-09-11T08:40:54.211 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-11T08:40:54.212 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-11T08:40:54.213 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T08:40:54.213 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T08:40:54.210+0000 7f3fc8c49640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 30899) UID: 1000 2024-09-11T08:40:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:40:54.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:40:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:40:54.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:54.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:54.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:40:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:40:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:40:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:54.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:40:54.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:54.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:40:54.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:54.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:40:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:40:54.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:40:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:40:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:55.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T08:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T08:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T08:40:55.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T08:40:55.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:55.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:55.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T08:40:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T08:40:55.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T08:40:55.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:55.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T08:40:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:55.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T08:40:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T08:40:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:55.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T08:40:55.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:55.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T08:40:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T08:40:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T08:40:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:55.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T08:40:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:55.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:40:55.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 4 2024-09-11T08:40:58.213 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-11T08:40:58.213 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:40:58.214 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-11T08:40:58.214 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-11T08:40:58.214 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-11T08:40:58.217 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-11T08:40:58.217 INFO:tasks.rbd_mirror_thrash:waiting for 94 secs before thrashing 2024-09-11T08:40:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:40:59.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:40:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:40:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:40:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:59.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:40:59.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:59.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:40:59.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:40:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:40:59.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:40:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:59.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:40:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:40:59.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:40:59.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:59.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:40:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:40:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:40:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:40:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:40:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:40:59.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:40:59.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:59.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:40:59.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T08:40:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T08:40:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T08:40:59.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T08:40:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:40:59.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T08:40:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:59.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:40:59.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:40:59.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T08:40:59.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:40:59.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T08:40:59.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T08:40:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:40:59.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T08:40:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:40:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:40:59.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:40:59.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:40:59.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 8 2024-09-11T08:41:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:41:07.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:41:07.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:41:07.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:41:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:41:07.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:41:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:41:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:41:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:41:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:41:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:41:07.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:41:07.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:41:07.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:41:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:41:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:41:07.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:41:07.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:41:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:41:07.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:41:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:41:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:41:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:41:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:41:07.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:41:07.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:41:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:41:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:41:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:41:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:41:07.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:41:07.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:41:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:41:07.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:41:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:41:07.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:41:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:41:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:41:08.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:41:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:41:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/test 2024-09-11T08:41:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T08:41:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:41:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:41:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T08:41:08.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:41:08.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T08:41:08.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T08:41:08.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T08:41:08.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:41:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:41:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T08:41:08.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/test 2024-09-11T08:41:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:41:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:41:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/test 2024-09-11T08:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T08:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:41:08.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T08:41:08.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:41:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T08:41:08.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T08:41:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T08:41:08.213 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T08:41:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:41:08.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:41:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T08:41:08.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/test 2024-09-11T08:41:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:41:08.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:41:08.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:41:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:41:08.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 8 2024-09-11T08:41:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:41:16.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:41:16.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:41:16.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:41:16.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:41:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:41:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:41:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:41:16.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:41:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:41:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:41:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:41:16.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:41:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:41:16.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:41:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:41:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:41:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name= 2024-09-11T08:41:16.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 10 2024-09-11T08:41:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:41:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:41:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:41:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:16.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:41:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=11 2024-09-11T08:41:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:41:16.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:41:16.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 45193216 bytes 2024-09-11T08:41:16.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 85071872 bytes 2024-09-11T08:41:16.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 113662976 bytes 2024-09-11T08:41:17.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 173133824 bytes 2024-09-11T08:41:17.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 218961920 bytes 2024-09-11T08:41:17.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 257199104 bytes 2024-09-11T08:41:17.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 285847552 bytes 2024-09-11T08:41:17.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 334354432 bytes 2024-09-11T08:41:17.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 374232064 bytes 2024-09-11T08:41:17.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 398474240 bytes 2024-09-11T08:41:17.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 403655680 bytes 2024-09-11T08:41:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 436527104 bytes 2024-09-11T08:41:18.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 474900480 bytes 2024-09-11T08:41:18.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 475526144 bytes 2024-09-11T08:41:18.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 499500032 bytes 2024-09-11T08:41:18.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 521950208 bytes 2024-09-11T08:41:18.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 565704704 bytes 2024-09-11T08:41:18.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 597956608 bytes 2024-09-11T08:41:18.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 669358080 bytes 2024-09-11T08:41:19.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 782820352 bytes 2024-09-11T08:41:19.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 857143296 bytes 2024-09-11T08:41:19.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 878003200 bytes 2024-09-11T08:41:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 970757120 bytes 2024-09-11T08:41:19.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1010970624 bytes 2024-09-11T08:41:20.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1072364544 bytes 2024-09-11T08:41:20.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1126339584 bytes 2024-09-11T08:41:20.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1140358144 bytes 2024-09-11T08:41:20.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1209036800 bytes 2024-09-11T08:41:20.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1250238464 bytes 2024-09-11T08:41:20.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1292157952 bytes 2024-09-11T08:41:20.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1331441664 bytes 2024-09-11T08:41:21.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1352714240 bytes 2024-09-11T08:41:21.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1384117248 bytes 2024-09-11T08:41:21.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1446595584 bytes 2024-09-11T08:41:21.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1539586048 bytes 2024-09-11T08:41:21.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1563070464 bytes 2024-09-11T08:41:21.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1599868928 bytes 2024-09-11T08:41:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1617414144 bytes 2024-09-11T08:41:22.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1687900160 bytes 2024-09-11T08:41:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1723551744 bytes 2024-09-11T08:41:22.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1793355776 bytes 2024-09-11T08:41:22.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1811388416 bytes 2024-09-11T08:41:22.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1811990528 bytes 2024-09-11T08:41:22.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1812574208 bytes 2024-09-11T08:41:22.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1863706624 bytes 2024-09-11T08:41:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1937870848 bytes 2024-09-11T08:41:23.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1978550272 bytes 2024-09-11T08:41:23.457 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:41:23.453+0000 7f00d11de640 -1 reset not still connected to 0x55b5dc05b040 2024-09-11T08:41:23.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2003058688 bytes 2024-09-11T08:41:23.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2051436544 bytes 2024-09-11T08:41:23.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2138716160 bytes 2024-09-11T08:41:23.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2155426816 bytes 2024-09-11T08:41:24.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2254386176 bytes 2024-09-11T08:41:24.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2265287680 bytes 2024-09-11T08:41:24.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2284748800 bytes 2024-09-11T08:41:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2322099200 bytes 2024-09-11T08:41:24.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2390380544 bytes 2024-09-11T08:41:24.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2430944256 bytes 2024-09-11T08:41:25.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2476924928 bytes 2024-09-11T08:41:25.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2536798208 bytes 2024-09-11T08:41:25.211 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:41:25.211 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:41:25.211 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T08:41:25.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:41:25.209+0000 7fc3956de640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 30794) UID: 1000 2024-09-11T08:41:25.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2555640832 bytes 2024-09-11T08:41:25.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2556237824 bytes 2024-09-11T08:41:25.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2569189376 bytes 2024-09-11T08:41:25.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2638431232 bytes 2024-09-11T08:41:25.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2720640000 bytes 2024-09-11T08:41:26.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2743406592 bytes 2024-09-11T08:41:26.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2795700224 bytes 2024-09-11T08:41:26.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2826043392 bytes 2024-09-11T08:41:26.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2872420352 bytes 2024-09-11T08:41:26.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2887473152 bytes 2024-09-11T08:41:26.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2964281344 bytes 2024-09-11T08:41:26.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 3008898048 bytes 2024-09-11T08:41:27.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 3046917120 bytes 2024-09-11T08:41:27.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 3106666496 bytes 2024-09-11T08:41:27.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 3172500480 bytes 2024-09-11T08:41:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:41:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:41:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:41:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:41:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:41:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:27.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:41:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:41:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:41:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:41:28.212 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:41:28.212 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:41:28.212 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:41:28.212 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:41:28.213 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:41:28.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:41:28.216 INFO:tasks.rbd_mirror_thrash:waiting for 105 secs before thrashing 2024-09-11T08:41:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:41:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:28.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:41:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:41:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:41:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:41:28.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:41:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: down+unknown 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: status not found 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:28' 2024-09-11T08:41:28.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:41:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 1 2024-09-11T08:41:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:41:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 2 2024-09-11T08:41:30.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:41:30.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: down+unknown 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: status not found 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:41:30.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:41:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:28' 2024-09-11T08:41:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:41:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 1 2024-09-11T08:41:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:41:30.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 4 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:41:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:41:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:41:34.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: down+unknown 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: status not found 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:41:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:41:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:28' 2024-09-11T08:41:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:41:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 1 2024-09-11T08:41:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:41:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 8 2024-09-11T08:41:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:41:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:41:42.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:41:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:41:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:41:42.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:41:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:41:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:41:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: down+unknown 2024-09-11T08:41:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: status not found 2024-09-11T08:41:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11T08:41:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:41:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:41:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:41:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:41:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:28' 2024-09-11T08:41:42.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:41:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 1 2024-09-11T08:41:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:41:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 8 2024-09-11T08:41:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:41:50.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:41:50.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: down+unknown 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: status not found 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:41:50.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:28' 2024-09-11T08:41:50.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:41:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 1 2024-09-11T08:41:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:41:50.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 8 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:41:58.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:41:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:41:58.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:41:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:41:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:41:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:41:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726044049,"remote_snapshot_timestamp":1726044049,"replay_state":"idle"} 2024-09-11T08:41:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:41:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:54 2024-09-11T08:41:58.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:41:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:41:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:41:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:41:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:58' 2024-09-11T08:41:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:41:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:41:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:41:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726044049,"remote_snapshot_timestamp":1726044049,"replay_state":"idle"} 2024-09-11T08:41:58.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:41:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:41:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:41:58.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:41:58.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:41:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:41:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:41:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:41:58.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:41:54' 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:41:54' 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:41:54' 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:41:58.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:41:59.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42174272mirror.2smithi07719.1.1-298-gb53b6d3efalseOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK43094312mirror.0smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":0.0,"last_snapshot_bytes":0,"last_snapshot_sync_seconds":0,"local_snapshot_timestamp":1726044049,"remote_snapshot_timestamp":1726044049,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:41:54cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:41:58' 2024-09-11T08:41:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:41:59.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:41:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:41:54' 2024-09-11T08:41:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:41:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:41:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:41:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:41:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:41:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:41:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:41:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap1 2024-09-11T08:41:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap1 2024-09-11T08:41:59.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:41:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:41:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:41:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap1 2024-09-11T08:41:59.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:42:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:42:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:00.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:42:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:00.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:42:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:42:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:42:00.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:42:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:42:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:01.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:42:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:42:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:42:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:42:01.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:42:01.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:42:01.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:42:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:42:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:42:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:42:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:42:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:42:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:42:01.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:42:01.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:42:01.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:42:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:42:01.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:42:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:42:01.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:42:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:42:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:42:01.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:42:01.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:42:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:42:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:42:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:42:01.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:42:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:42:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:42:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:42:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:42:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:42:01.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:42:01.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:42:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:42:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:42:01.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:42:01.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:42:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:42:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:42:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:42:01.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:42:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:42:01.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:01.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:42:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:42:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:42:01.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:42:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:42:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:42:01.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:42:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:42:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:01.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:42:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 5 2024-09-11T08:42:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:42:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:02.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:42:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:42:02.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:02.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:42:02.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=5 2024-09-11T08:42:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:42:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:02.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:42:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:02.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:02.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:02.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=3 2024-09-11T08:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 5 = 3 2024-09-11T08:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:02.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:42:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:03.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:03.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:03.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:42:03.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:42:03.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:42:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:42:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 5 = '' 2024-09-11T08:42:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:03.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:42:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:03.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:42:04.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:42:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:42:04.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:42:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:42:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 5 = '' 2024-09-11T08:42:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:04.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:42:05.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:05.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:05.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=5 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 5 = 5 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap1 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap1 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:42:05.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:42:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap1 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:42:06.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:42:06.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:42:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:06.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap1 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap1 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:42:06.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:42:07.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:42:07.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:42:08.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:42:08.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:42:08.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:42:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:09.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap1-clean 2024-09-11T08:42:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap1-clean 2024-09-11T08:42:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap1-clean 2024-09-11T08:42:09.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap1-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:42:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:42:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:42:09.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:09.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:42:09.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=13 2024-09-11T08:42:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:42:09.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:42:09.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 66150400 bytes 2024-09-11T08:42:09.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 147313664 bytes 2024-09-11T08:42:09.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 174928896 bytes 2024-09-11T08:42:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 210467840 bytes 2024-09-11T08:42:10.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 230226944 bytes 2024-09-11T08:42:10.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 255783936 bytes 2024-09-11T08:42:10.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 312702976 bytes 2024-09-11T08:42:10.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 336115712 bytes 2024-09-11T08:42:10.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 356001792 bytes 2024-09-11T08:42:10.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 396013568 bytes 2024-09-11T08:42:10.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 420873216 bytes 2024-09-11T08:42:11.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 456231936 bytes 2024-09-11T08:42:11.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 470168576 bytes 2024-09-11T08:42:11.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 481285120 bytes 2024-09-11T08:42:11.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 501580800 bytes 2024-09-11T08:42:11.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 551797760 bytes 2024-09-11T08:42:11.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 555724800 bytes 2024-09-11T08:42:11.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 590334976 bytes 2024-09-11T08:42:11.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 626027520 bytes 2024-09-11T08:42:12.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 676487168 bytes 2024-09-11T08:42:12.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 733710336 bytes 2024-09-11T08:42:12.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 866786304 bytes 2024-09-11T08:42:12.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 952552448 bytes 2024-09-11T08:42:12.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 967983104 bytes 2024-09-11T08:42:13.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1040201728 bytes 2024-09-11T08:42:13.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1048308736 bytes 2024-09-11T08:42:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1087366144 bytes 2024-09-11T08:42:13.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1152870400 bytes 2024-09-11T08:42:13.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1189549056 bytes 2024-09-11T08:42:13.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1214857216 bytes 2024-09-11T08:42:13.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1230295040 bytes 2024-09-11T08:42:14.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1247104000 bytes 2024-09-11T08:42:14.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1272670208 bytes 2024-09-11T08:42:14.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1308788736 bytes 2024-09-11T08:42:14.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1363530752 bytes 2024-09-11T08:42:14.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1429771264 bytes 2024-09-11T08:42:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1513860096 bytes 2024-09-11T08:42:15.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1567060992 bytes 2024-09-11T08:42:15.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1599255552 bytes 2024-09-11T08:42:15.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1613527040 bytes 2024-09-11T08:42:15.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1661216768 bytes 2024-09-11T08:42:15.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1745009664 bytes 2024-09-11T08:42:16.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1783088128 bytes 2024-09-11T08:42:16.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1828833280 bytes 2024-09-11T08:42:16.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1888323584 bytes 2024-09-11T08:42:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1958788096 bytes 2024-09-11T08:42:16.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1959398400 bytes 2024-09-11T08:42:16.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1978456064 bytes 2024-09-11T08:42:17.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2042014720 bytes 2024-09-11T08:42:17.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2073660416 bytes 2024-09-11T08:42:17.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2113757184 bytes 2024-09-11T08:42:17.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2120028160 bytes 2024-09-11T08:42:17.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2147342336 bytes 2024-09-11T08:42:17.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2153168896 bytes 2024-09-11T08:42:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2184919040 bytes 2024-09-11T08:42:17.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2222977024 bytes 2024-09-11T08:42:18.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2281510912 bytes 2024-09-11T08:42:18.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2305396736 bytes 2024-09-11T08:42:18.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2345401344 bytes 2024-09-11T08:42:18.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2351853568 bytes 2024-09-11T08:42:18.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2360630272 bytes 2024-09-11T08:42:18.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2407550976 bytes 2024-09-11T08:42:18.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2454793216 bytes 2024-09-11T08:42:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2468514816 bytes 2024-09-11T08:42:19.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2478839808 bytes 2024-09-11T08:42:19.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2510044160 bytes 2024-09-11T08:42:19.253 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2532974592 bytes 2024-09-11T08:42:19.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2591646720 bytes 2024-09-11T08:42:19.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2663316480 bytes 2024-09-11T08:42:19.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2696107008 bytes 2024-09-11T08:42:19.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2751421440 bytes 2024-09-11T08:42:20.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2806214656 bytes 2024-09-11T08:42:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 2831223808 bytes 2024-09-11T08:42:20.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 2870416384 bytes 2024-09-11T08:42:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 2911312896 bytes 2024-09-11T08:42:20.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 2963405824 bytes 2024-09-11T08:42:20.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7700 IOs, 3018293248 bytes 2024-09-11T08:42:21.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7800 IOs, 3060220928 bytes 2024-09-11T08:42:21.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7900 IOs, 3115752448 bytes 2024-09-11T08:42:21.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8000 IOs, 3187116032 bytes 2024-09-11T08:42:21.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8100 IOs, 3225756672 bytes 2024-09-11T08:42:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8200 IOs, 3298483200 bytes 2024-09-11T08:42:21.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8300 IOs, 3299094528 bytes 2024-09-11T08:42:21.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8400 IOs, 3321462784 bytes 2024-09-11T08:42:22.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8500 IOs, 3374266368 bytes 2024-09-11T08:42:22.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8600 IOs, 3426877440 bytes 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:42:22.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:42:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:42:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:42:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:42:23.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:42:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:42:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:42:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:42:23.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:42:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:42:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:42:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:42:23.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":44246016.0,"bytes_per_snapshot":530153472.0,"last_snapshot_bytes":530153472,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044122,"remote_snapshot_timestamp":1726044122,"replay_state":"idle"} 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:42:06 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:41:58' 2024-09-11T08:42:23.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:42:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:42:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:42:23.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":44246016.0,"bytes_per_snapshot":530153472.0,"last_snapshot_bytes":530153472,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044122,"remote_snapshot_timestamp":1726044122,"replay_state":"idle"} 2024-09-11T08:42:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:42:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:42:23.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:42:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:42:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:42:23.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:42:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:42:23.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:42:06' 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:42:06' 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:42:06' 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:42:23.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42174272mirror.2smithi07719.1.1-298-gb53b6d3efalseOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK43094312mirror.0smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":44246016.0,"bytes_per_snapshot":530153472.0,"last_snapshot_bytes":530153472,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044122,"remote_snapshot_timestamp":1726044122,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:42:06cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:41:58' 2024-09-11T08:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:42:23.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:42:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:42:06' 2024-09-11T08:42:23.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:42:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:42:23.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:42:23.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:42:23.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap2 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap2 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap2 2024-09-11T08:42:23.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:42:32.218 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T08:42:32.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T08:42:32.219 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-11T08:42:32.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T08:42:32.220+0000 7f0cc1c4f640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 30858) UID: 1000 2024-09-11T08:42:33.219 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T08:42:33.220 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:42:33.220 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T08:42:33.220 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T08:42:33.221 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T08:42:33.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T08:42:33.224 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-11T08:42:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:42:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:42:53.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:42:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:42:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:42:53.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:42:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:42:54.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:42:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:42:54.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:42:54.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:42:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:42:54.407 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:42:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:42:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:42:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:42:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:42:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:42:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:42:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:42:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:42:54.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:42:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:42:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:42:54.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:42:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:42:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:42:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:42:54.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:42:54.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:42:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:42:54.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:42:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:42:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:42:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:42:54.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:42:54.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:42:54.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:42:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:42:54.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:42:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:42:54.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:42:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:42:54.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:42:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:42:54.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:42:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:42:54.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:42:54.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:42:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:42:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:42:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:42:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:42:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:54.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:42:54.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:42:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:42:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:42:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:42:54.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:42:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:42:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:54.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:42:56.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 8 2024-09-11T08:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:42:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:42:56.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:56.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:42:56.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=8 2024-09-11T08:42:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:42:56.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:56.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:56.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:56.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:42:56.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:42:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:42:56.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:42:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:42:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 8 = '' 2024-09-11T08:42:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:56.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:57.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:42:57.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:42:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:42:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:42:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:42:57.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 8 = '' 2024-09-11T08:42:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:57.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:42:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:58.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:58.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:42:58.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:42:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:42:58.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:42:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:42:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 8 = '' 2024-09-11T08:42:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:42:58.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:42:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:59.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:42:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:42:59.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:42:59.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:42:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=8 2024-09-11T08:42:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 8 = 8 2024-09-11T08:42:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:42:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap2 2024-09-11T08:42:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:42:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:42:59.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:42:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap2 2024-09-11T08:42:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:42:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:42:59.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:43:00.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap2 2024-09-11T08:43:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 4 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:43:00.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:43:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:43:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:43:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:43:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:43:00.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:43:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap1-clean ']' 2024-09-11T08:43:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-11T08:43:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:00.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap1-clean 2024-09-11T08:43:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:43:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:01.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:01.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:02.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:02.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:02.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap2 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap2 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:03.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:03.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:04.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:04.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:05.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:05.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap2-clean 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap2-clean 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap2-clean 2024-09-11T08:43:05.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap2-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:43:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:43:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:43:06.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:06.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:43:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:43:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:43:06.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:43:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 69436416 bytes 2024-09-11T08:43:06.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 112173056 bytes 2024-09-11T08:43:07.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 150375424 bytes 2024-09-11T08:43:07.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 191885312 bytes 2024-09-11T08:43:07.358 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 230216704 bytes 2024-09-11T08:43:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 268721152 bytes 2024-09-11T08:43:07.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 330772480 bytes 2024-09-11T08:43:07.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 377922560 bytes 2024-09-11T08:43:08.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 413335552 bytes 2024-09-11T08:43:08.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 462264320 bytes 2024-09-11T08:43:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 486934528 bytes 2024-09-11T08:43:08.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 508474368 bytes 2024-09-11T08:43:08.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 522484736 bytes 2024-09-11T08:43:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 528887808 bytes 2024-09-11T08:43:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 610312192 bytes 2024-09-11T08:43:09.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 705569792 bytes 2024-09-11T08:43:09.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 706193408 bytes 2024-09-11T08:43:09.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 723231744 bytes 2024-09-11T08:43:09.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 793791488 bytes 2024-09-11T08:43:09.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 829254656 bytes 2024-09-11T08:43:09.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 829908992 bytes 2024-09-11T08:43:09.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 879305728 bytes 2024-09-11T08:43:10.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 909496320 bytes 2024-09-11T08:43:10.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 945732608 bytes 2024-09-11T08:43:10.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 982033408 bytes 2024-09-11T08:43:10.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1001926656 bytes 2024-09-11T08:43:10.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1053150208 bytes 2024-09-11T08:43:10.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1105418240 bytes 2024-09-11T08:43:11.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1158190080 bytes 2024-09-11T08:43:11.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1186238464 bytes 2024-09-11T08:43:11.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1219257344 bytes 2024-09-11T08:43:11.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1268254720 bytes 2024-09-11T08:43:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1283739648 bytes 2024-09-11T08:43:11.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1326320640 bytes 2024-09-11T08:43:11.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1339587584 bytes 2024-09-11T08:43:11.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1349379072 bytes 2024-09-11T08:43:12.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1391844352 bytes 2024-09-11T08:43:12.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1423442944 bytes 2024-09-11T08:43:12.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1450140672 bytes 2024-09-11T08:43:12.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1509667840 bytes 2024-09-11T08:43:12.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1604231168 bytes 2024-09-11T08:43:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1623279616 bytes 2024-09-11T08:43:13.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1678032896 bytes 2024-09-11T08:43:13.217 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:43:13.217 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:43:13.217 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T08:43:13.218 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:43:13.216+0000 7f707575a640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 32322) UID: 1000 2024-09-11T08:43:13.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1751918592 bytes 2024-09-11T08:43:13.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1805295616 bytes 2024-09-11T08:43:13.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1824509952 bytes 2024-09-11T08:43:13.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1825096704 bytes 2024-09-11T08:43:13.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1859121152 bytes 2024-09-11T08:43:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1879488512 bytes 2024-09-11T08:43:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1942830080 bytes 2024-09-11T08:43:14.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 1972072448 bytes 2024-09-11T08:43:14.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 1981959168 bytes 2024-09-11T08:43:14.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2067533824 bytes 2024-09-11T08:43:14.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2112271360 bytes 2024-09-11T08:43:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2165616640 bytes 2024-09-11T08:43:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2211304448 bytes 2024-09-11T08:43:15.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2281481216 bytes 2024-09-11T08:43:15.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2327882752 bytes 2024-09-11T08:43:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:43:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:43:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:43:15.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:43:15.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:43:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:43:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:16.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:43:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:43:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:43:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:43:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:43:16.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:43:16.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":17879688.53,"bytes_per_snapshot":533272064.0,"last_snapshot_bytes":536390656,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044175,"remote_snapshot_timestamp":1726044175,"replay_state":"idle"} 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:43:06 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:42:58' 2024-09-11T08:43:16.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:43:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:43:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:43:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":17879688.53,"bytes_per_snapshot":533272064.0,"last_snapshot_bytes":536390656,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044175,"remote_snapshot_timestamp":1726044175,"replay_state":"idle"} 2024-09-11T08:43:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:43:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:43:16.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:43:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:43:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:43:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:43:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:43:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:43:06' 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:43:06' 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:43:06' 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:43:16.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:43:16.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:43:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42174272mirror.2smithi07719.1.1-298-gb53b6d3efalseOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":17879688.53,"bytes_per_snapshot":533272064.0,"last_snapshot_bytes":536390656,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044175,"remote_snapshot_timestamp":1726044175,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:43:06cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:42:58' 2024-09-11T08:43:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:43:16.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:43:16.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:43:06' 2024-09-11T08:43:16.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:43:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:43:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:43:16.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:43:16.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:16.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap3 2024-09-11T08:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap3 2024-09-11T08:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap3 2024-09-11T08:43:16.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:43:21.218 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:43:21.219 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:43:21.219 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:43:21.219 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:43:21.219 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:43:21.223 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:43:21.223 INFO:tasks.rbd_mirror_thrash:waiting for 77 secs before thrashing 2024-09-11T08:43:40.224 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T08:43:40.225 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T08:43:40.225 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-11T08:43:40.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T08:43:40.227+0000 7ff630d84640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 32953) UID: 1000 2024-09-11T08:43:45.226 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T08:43:45.226 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:43:45.226 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T08:43:45.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T08:43:45.227 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T08:43:45.230 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T08:43:45.230 INFO:tasks.rbd_mirror_thrash:waiting for 64 secs before thrashing 2024-09-11T08:43:45.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:43:45.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:43:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:43:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:43:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:43:46.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:43:46.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:43:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:43:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:43:46.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:43:46.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:43:46.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:43:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:43:46.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:43:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:43:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:43:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:43:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:43:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:43:46.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:43:46.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:43:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:43:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:43:46.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:43:46.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:43:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:43:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:43:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:43:46.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:43:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:43:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:43:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:43:46.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:43:46.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:43:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:43:46.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:43:46.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:43:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:43:46.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:43:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:43:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:43:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:43:46.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:43:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:43:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:43:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:43:46.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:43:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:46.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:43:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:46.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:43:48.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 11 2024-09-11T08:43:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:43:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:43:48.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:43:48.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:43:48.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:43:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=11 2024-09-11T08:43:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:43:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:43:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:43:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:43:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:43:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:43:48.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:43:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:43:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:43:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:43:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:43:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 11 = '' 2024-09-11T08:43:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:43:48.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:43:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:49.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:43:49.358 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:43:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:43:49.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:43:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:43:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:43:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:43:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:43:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 11 = '' 2024-09-11T08:43:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:43:49.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:43:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:50.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:50.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:43:50.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:43:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:43:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:43:50.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:43:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:43:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:43:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:43:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 11 = '' 2024-09-11T08:43:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:43:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:43:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:43:51.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:43:51.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:43:51.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:43:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=11 2024-09-11T08:43:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 11 = 11 2024-09-11T08:43:51.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap3 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap3 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:43:51.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:43:52.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap3 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 6 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:43:53.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:43:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:43:01 2024 2024-09-11T08:43:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:43:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:43:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:43:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:43:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:43:53.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:43:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap2-clean ']' 2024-09-11T08:43:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-11T08:43:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap2-clean 2024-09-11T08:43:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:43:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:53.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:53.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:53.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:54.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:54.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:55.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap3 2024-09-11T08:43:55.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap3 2024-09-11T08:43:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:43:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:55.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:43:55.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:55.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:57.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:43:57.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:43:57.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:43:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap3-clean 2024-09-11T08:43:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap3-clean 2024-09-11T08:43:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap3-clean 2024-09-11T08:43:57.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap3-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:43:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:43:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:43:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:43:58.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:43:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:43:58.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:43:58.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:43:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:43:58.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:43:58.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 23670784 bytes 2024-09-11T08:43:58.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 41278464 bytes 2024-09-11T08:43:59.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 136579072 bytes 2024-09-11T08:43:59.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 181641216 bytes 2024-09-11T08:43:59.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 203521024 bytes 2024-09-11T08:43:59.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 204126208 bytes 2024-09-11T08:43:59.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 271544320 bytes 2024-09-11T08:43:59.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 288999424 bytes 2024-09-11T08:43:59.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 327601152 bytes 2024-09-11T08:44:00.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 337486848 bytes 2024-09-11T08:44:00.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 405798912 bytes 2024-09-11T08:44:00.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 458063872 bytes 2024-09-11T08:44:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 497709056 bytes 2024-09-11T08:44:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 542818304 bytes 2024-09-11T08:44:00.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 570368000 bytes 2024-09-11T08:44:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 610810880 bytes 2024-09-11T08:44:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 638158848 bytes 2024-09-11T08:44:01.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 711861248 bytes 2024-09-11T08:44:01.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 741637120 bytes 2024-09-11T08:44:01.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 764070912 bytes 2024-09-11T08:44:01.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 786453504 bytes 2024-09-11T08:44:01.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 854767616 bytes 2024-09-11T08:44:02.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 918745088 bytes 2024-09-11T08:44:02.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 975689728 bytes 2024-09-11T08:44:02.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1028988928 bytes 2024-09-11T08:44:02.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1083171840 bytes 2024-09-11T08:44:02.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1113299968 bytes 2024-09-11T08:44:02.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1185790976 bytes 2024-09-11T08:44:03.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1292112896 bytes 2024-09-11T08:44:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1358522368 bytes 2024-09-11T08:44:03.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1388861440 bytes 2024-09-11T08:44:03.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1431857152 bytes 2024-09-11T08:44:03.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1450687488 bytes 2024-09-11T08:44:03.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1481571328 bytes 2024-09-11T08:44:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1491653632 bytes 2024-09-11T08:44:04.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1492262912 bytes 2024-09-11T08:44:04.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1540518912 bytes 2024-09-11T08:44:04.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1589637120 bytes 2024-09-11T08:44:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1671759872 bytes 2024-09-11T08:44:04.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1733407744 bytes 2024-09-11T08:44:05.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1796250624 bytes 2024-09-11T08:44:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1857624064 bytes 2024-09-11T08:44:05.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1901132800 bytes 2024-09-11T08:44:05.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1901771776 bytes 2024-09-11T08:44:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:44:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:44:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:44:05.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:44:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:44:05.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:44:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:44:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:44:05.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:44:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:44:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:44:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:44:06.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:44:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:44:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:44:06.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:44:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:44:06.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":17659972.27,"bytes_per_snapshot":533094912.0,"last_snapshot_bytes":529799168,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044227,"remote_snapshot_timestamp":1726044227,"replay_state":"idle"} 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:44:06 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:43:58' 2024-09-11T08:44:06.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:44:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:44:06.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:44:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":17659972.27,"bytes_per_snapshot":533094912.0,"last_snapshot_bytes":529799168,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044227,"remote_snapshot_timestamp":1726044227,"replay_state":"idle"} 2024-09-11T08:44:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:44:06.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:44:06.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:44:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:44:06.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:44:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:44:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:44:06.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:44:06' 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:44:06' 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:44:06' 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:44:06.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:44:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42174272mirror.2smithi07719.1.1-298-gb53b6d3efalseOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK43974400mirror.0smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":17659972.27,"bytes_per_snapshot":533094912.0,"last_snapshot_bytes":529799168,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044227,"remote_snapshot_timestamp":1726044227,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:44:06cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:43:58' 2024-09-11T08:44:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:44:06.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:44:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:44:06' 2024-09-11T08:44:06.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:44:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:44:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:44:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:44:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:44:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap4 2024-09-11T08:44:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap4 2024-09-11T08:44:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:44:06.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap4 2024-09-11T08:44:06.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:44:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:44:34.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:44:34.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:44:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:44:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:44:35.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:44:35.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:44:35.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:44:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:44:35.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:44:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:44:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:44:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:44:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:44:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:44:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:44:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:44:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:44:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:44:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:44:35.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:44:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:44:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:44:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:44:35.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:44:35.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:44:35.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:44:35.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:44:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:44:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:44:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:44:35.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:44:36.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:44:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:44:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:44:36.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:44:36.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:44:36.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:44:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:44:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:44:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:44:36.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:44:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:44:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:44:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:44:36.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:44:36.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:44:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:44:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:36.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:44:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 14 2024-09-11T08:44:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:44:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:44:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:44:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:44:37.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:44:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:44:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=14 2024-09-11T08:44:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:44:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:44:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:44:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:38.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:44:38.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:44:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:44:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:44:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:44:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:44:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:44:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:44:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 14 = '' 2024-09-11T08:44:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:44:38.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:44:38.223 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7928713920710442) > weight (0.25) 2024-09-11T08:44:38.224 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.38303388413029793) > weight (0.25) 2024-09-11T08:44:38.224 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.2671472004578218) > weight (0.25) 2024-09-11T08:44:38.224 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.3209910754380828) > weight (0.25) 2024-09-11T08:44:38.224 INFO:tasks.rbd_mirror_thrash:waiting for 87 secs before thrashing 2024-09-11T08:44:38.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:38.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:44:38.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:44:38.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:44:38.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:44:38.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 14 = '' 2024-09-11T08:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:44:38.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:44:39.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:44:39.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:44:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:44:39.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:44:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:44:39.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:44:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:44:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:44:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 14 = '' 2024-09-11T08:44:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:44:39.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:44:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:44:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:44:41.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:44:41.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:44:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=14 2024-09-11T08:44:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 14 = 14 2024-09-11T08:44:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap4 2024-09-11T08:44:41.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:44:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap4 2024-09-11T08:44:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:44:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:44:41.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:44:42.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap4 2024-09-11T08:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:44:42.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 8 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:43:01 2024 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:44:42.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:44:42.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:44:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap3-clean ']' 2024-09-11T08:44:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-11T08:44:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap3-clean 2024-09-11T08:44:42.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:44:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:44:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:42.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:44:42.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:44:42.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:43.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:44:44.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:44.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:44.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap4 2024-09-11T08:44:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap4 2024-09-11T08:44:44.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:44:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:44:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:44.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:44.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:44:45.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:44:45.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:46.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:44:46.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:46.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:44:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:44:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap4-clean 2024-09-11T08:44:46.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap4-clean 2024-09-11T08:44:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:44:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap4-clean 2024-09-11T08:44:46.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap4-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:44:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:44:47.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:44:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:44:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:44:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:44:47.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:44:47.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=14 2024-09-11T08:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:44:47.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:44:47.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 30694400 bytes 2024-09-11T08:44:47.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 109026304 bytes 2024-09-11T08:44:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 188665856 bytes 2024-09-11T08:44:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 223143936 bytes 2024-09-11T08:44:48.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 305061888 bytes 2024-09-11T08:44:48.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 334615552 bytes 2024-09-11T08:44:48.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 389386240 bytes 2024-09-11T08:44:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 422206464 bytes 2024-09-11T08:44:49.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 422834176 bytes 2024-09-11T08:44:49.231 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7154025209101255) > weight (0.25) 2024-09-11T08:44:49.231 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.8126488667896815) > weight (0.25) 2024-09-11T08:44:49.231 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.6385923903111184) > weight (0.25) 2024-09-11T08:44:49.231 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.5148138793795591) > weight (0.25) 2024-09-11T08:44:49.231 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-09-11T08:44:49.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 499395584 bytes 2024-09-11T08:44:49.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 538195968 bytes 2024-09-11T08:44:49.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 583121920 bytes 2024-09-11T08:44:49.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 633521152 bytes 2024-09-11T08:44:49.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 677855232 bytes 2024-09-11T08:44:50.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 701759488 bytes 2024-09-11T08:44:50.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 746643456 bytes 2024-09-11T08:44:50.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 828147712 bytes 2024-09-11T08:44:50.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 869843968 bytes 2024-09-11T08:44:50.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 973430784 bytes 2024-09-11T08:44:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1010833408 bytes 2024-09-11T08:44:51.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1038396416 bytes 2024-09-11T08:44:51.359 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1102939136 bytes 2024-09-11T08:44:51.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1141238784 bytes 2024-09-11T08:44:51.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1147488256 bytes 2024-09-11T08:44:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1199915008 bytes 2024-09-11T08:44:51.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1292698624 bytes 2024-09-11T08:44:52.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1324913664 bytes 2024-09-11T08:44:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1367349248 bytes 2024-09-11T08:44:52.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1371415552 bytes 2024-09-11T08:44:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1406061568 bytes 2024-09-11T08:44:52.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1439409152 bytes 2024-09-11T08:44:52.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1492364288 bytes 2024-09-11T08:44:53.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1574094848 bytes 2024-09-11T08:44:53.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1587813376 bytes 2024-09-11T08:44:53.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1635675136 bytes 2024-09-11T08:44:53.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1659449344 bytes 2024-09-11T08:44:53.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1694521344 bytes 2024-09-11T08:44:53.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1743125504 bytes 2024-09-11T08:44:53.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1767961600 bytes 2024-09-11T08:44:54.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1809470464 bytes 2024-09-11T08:44:54.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1832502272 bytes 2024-09-11T08:44:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1940043776 bytes 2024-09-11T08:44:54.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1940647936 bytes 2024-09-11T08:44:54.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 2000164864 bytes 2024-09-11T08:44:54.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2053427200 bytes 2024-09-11T08:44:54.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2079013888 bytes 2024-09-11T08:44:55.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2079635456 bytes 2024-09-11T08:44:55.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2144328704 bytes 2024-09-11T08:44:55.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2172987392 bytes 2024-09-11T08:44:55.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2197637120 bytes 2024-09-11T08:44:55.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2251190272 bytes 2024-09-11T08:44:55.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2278697984 bytes 2024-09-11T08:44:55.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2323922944 bytes 2024-09-11T08:44:56.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2370145280 bytes 2024-09-11T08:44:56.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2422695936 bytes 2024-09-11T08:44:56.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2491593728 bytes 2024-09-11T08:44:56.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2562357248 bytes 2024-09-11T08:44:57.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2593008640 bytes 2024-09-11T08:44:57.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2638003200 bytes 2024-09-11T08:44:57.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2691145728 bytes 2024-09-11T08:44:57.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2764014592 bytes 2024-09-11T08:44:57.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2765572096 bytes 2024-09-11T08:44:57.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2825410560 bytes 2024-09-11T08:44:58.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2843287552 bytes 2024-09-11T08:44:58.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2870551552 bytes 2024-09-11T08:44:58.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2899755008 bytes 2024-09-11T08:44:58.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2987584512 bytes 2024-09-11T08:44:58.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 3037133824 bytes 2024-09-11T08:44:58.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 3086397440 bytes 2024-09-11T08:44:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 3116236800 bytes 2024-09-11T08:44:59.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 3117416448 bytes 2024-09-11T08:44:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 3138942976 bytes 2024-09-11T08:44:59.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 3185828864 bytes 2024-09-11T08:44:59.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 3236939776 bytes 2024-09-11T08:44:59.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 3257572352 bytes 2024-09-11T08:44:59.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 3273575424 bytes 2024-09-11T08:45:00.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7700 IOs, 3307927552 bytes 2024-09-11T08:45:00.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7800 IOs, 3360661504 bytes 2024-09-11T08:45:00.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7900 IOs, 3388917760 bytes 2024-09-11T08:45:00.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8000 IOs, 3421716480 bytes 2024-09-11T08:45:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8100 IOs, 3441140736 bytes 2024-09-11T08:45:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8200 IOs, 3470846976 bytes 2024-09-11T08:45:00.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8300 IOs, 3479914496 bytes 2024-09-11T08:45:00.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8400 IOs, 3523183616 bytes 2024-09-11T08:45:01.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8500 IOs, 3613195264 bytes 2024-09-11T08:45:01.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8600 IOs, 3634869248 bytes 2024-09-11T08:45:01.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8700 IOs, 3676494848 bytes 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:01.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:45:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:45:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:45:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:45:01.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:45:02.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:45:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:45:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:45:02.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:45:02.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533094912.0,"last_snapshot_bytes":529799168,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044227,"remote_snapshot_timestamp":1726044227,"replay_state":"idle"} 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:44:36 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:44:58' 2024-09-11T08:45:02.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:45:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:45:02.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:45:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533094912.0,"last_snapshot_bytes":529799168,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044227,"remote_snapshot_timestamp":1726044227,"replay_state":"idle"} 2024-09-11T08:45:02.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:45:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:45:02.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:45:02.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:45:02.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:45:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:45:02.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:45:02.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:44:36' 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:44:36' 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:44:36' 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:45:02.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:45:02.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42174272mirror.2smithi07719.1.1-298-gb53b6d3efalseOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK43974400mirror.0smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533094912.0,"last_snapshot_bytes":529799168,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044227,"remote_snapshot_timestamp":1726044227,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:44:36cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:44:58' 2024-09-11T08:45:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:45:02.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:45:02.704 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:44:36' 2024-09-11T08:45:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:45:02.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:45:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:45:02.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:45:02.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:45:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:02.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap5 2024-09-11T08:45:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap5 2024-09-11T08:45:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap5 2024-09-11T08:45:02.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:45:31.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:45:31.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:45:31.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:45:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:45:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:45:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:45:32.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:45:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:45:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:45:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:45:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:45:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:45:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:45:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:45:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:45:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:45:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:45:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:45:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:45:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:45:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:45:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:45:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:45:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:45:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:45:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:45:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:45:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:45:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:45:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:45:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:45:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:45:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:45:32.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:45:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:45:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:45:32.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:45:32.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:45:32.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:45:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:45:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:45:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:45:32.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:45:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:45:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:45:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:45:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:45:32.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:45:32.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:45:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:45:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:45:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:45:32.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:45:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:45:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:45:32.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:45:32.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:45:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:45:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:45:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:45:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:45:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:45:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:32.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:45:32.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:45:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:45:32.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:45:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:45:32.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:45:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:45:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:32.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:32.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:45:34.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 17 2024-09-11T08:45:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:45:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:34.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:45:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:45:34.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:45:34.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:45:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=17 2024-09-11T08:45:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:45:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:45:34.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:45:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:34.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:45:34.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:45:35.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:45:35.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:45:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:45:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:45:35.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:45:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:45:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 17 = '' 2024-09-11T08:45:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:45:35.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:45:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:35.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:45:35.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:45:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:45:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:45:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:45:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:45:35.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:45:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:45:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 17 = '' 2024-09-11T08:45:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:45:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:45:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:45:36.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:45:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:45:36.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:45:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:45:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:45:36.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:45:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:45:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 17 = '' 2024-09-11T08:45:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:45:36.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:45:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:38.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:45:38.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:45:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:45:38.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=17 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 17 = 17 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap5 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap5 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:45:38.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:45:39.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap5 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 10 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:45:39.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:45:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:44:42 2024 2024-09-11T08:45:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:45:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:45:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:45:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:45:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:45:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap4-clean ']' 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap4-clean 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:45:39.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:45:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:45:39.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:45:39.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:41.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:45:41.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:41.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap5 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap5 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:41.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:45:42.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:45:42.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:43.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:45:43.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:45:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap5-clean 2024-09-11T08:45:43.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap5-clean 2024-09-11T08:45:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap5-clean 2024-09-11T08:45:43.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap5-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:45:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:45:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:45:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:44.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:45:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=8 2024-09-11T08:45:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:45:44.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:45:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 13931520 bytes 2024-09-11T08:45:44.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 38691840 bytes 2024-09-11T08:45:44.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 48644096 bytes 2024-09-11T08:45:45.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 56193024 bytes 2024-09-11T08:45:45.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 117602304 bytes 2024-09-11T08:45:45.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 171303936 bytes 2024-09-11T08:45:45.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 251457536 bytes 2024-09-11T08:45:45.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 385942528 bytes 2024-09-11T08:45:46.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 417476608 bytes 2024-09-11T08:45:46.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 418079744 bytes 2024-09-11T08:45:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 436470784 bytes 2024-09-11T08:45:46.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 487962624 bytes 2024-09-11T08:45:46.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 535370752 bytes 2024-09-11T08:45:46.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 604708864 bytes 2024-09-11T08:45:46.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 635719680 bytes 2024-09-11T08:45:47.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 676288512 bytes 2024-09-11T08:45:47.253 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 729035776 bytes 2024-09-11T08:45:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 753209344 bytes 2024-09-11T08:45:47.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 766464000 bytes 2024-09-11T08:45:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 823245824 bytes 2024-09-11T08:45:47.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 901588992 bytes 2024-09-11T08:45:48.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 934270976 bytes 2024-09-11T08:45:48.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 976208896 bytes 2024-09-11T08:45:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1012468736 bytes 2024-09-11T08:45:48.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1056397312 bytes 2024-09-11T08:45:48.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1075867648 bytes 2024-09-11T08:45:48.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1092081664 bytes 2024-09-11T08:45:48.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1134160896 bytes 2024-09-11T08:45:49.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1166840832 bytes 2024-09-11T08:45:49.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1208130560 bytes 2024-09-11T08:45:49.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1276682240 bytes 2024-09-11T08:45:49.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1277285376 bytes 2024-09-11T08:45:49.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1285502976 bytes 2024-09-11T08:45:49.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1370288128 bytes 2024-09-11T08:45:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1456508928 bytes 2024-09-11T08:45:50.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1546009600 bytes 2024-09-11T08:45:50.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1583596544 bytes 2024-09-11T08:45:50.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1640867840 bytes 2024-09-11T08:45:50.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1641463808 bytes 2024-09-11T08:45:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1712158720 bytes 2024-09-11T08:45:51.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1758775296 bytes 2024-09-11T08:45:51.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1785157632 bytes 2024-09-11T08:45:51.232 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.6781977370155928) > weight (0.25) 2024-09-11T08:45:51.232 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5562347588366936) > weight (0.25) 2024-09-11T08:45:51.232 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5394557026305384) > weight (0.25) 2024-09-11T08:45:51.232 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.9860166148233747) > weight (0.25) 2024-09-11T08:45:51.233 INFO:tasks.rbd_mirror_thrash:waiting for 62 secs before thrashing 2024-09-11T08:45:51.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1829587968 bytes 2024-09-11T08:45:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1898699776 bytes 2024-09-11T08:45:51.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1929475072 bytes 2024-09-11T08:45:51.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1990738944 bytes 2024-09-11T08:45:52.213 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2059352064 bytes 2024-09-11T08:45:52.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2152418304 bytes 2024-09-11T08:45:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:45:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:45:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:45:52.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:45:52.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:45:53.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:45:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:45:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:45:53.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":524428800.0,"last_snapshot_bytes":519058432,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044276,"remote_snapshot_timestamp":1726044333,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1726044333} 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:45:36 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:45:28' 2024-09-11T08:45:53.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:45:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:45:53.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:45:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":524428800.0,"last_snapshot_bytes":519058432,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044276,"remote_snapshot_timestamp":1726044333,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1726044333} 2024-09-11T08:45:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:45:53.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:45:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:45:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:45:53.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:45:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:45:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:45:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:45:36' 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:45:36' 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:45:36' 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:45:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:45:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42174272mirror.2smithi07719.1.1-298-gb53b6d3efalseOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK43974400mirror.0smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":524428800.0,"last_snapshot_bytes":519058432,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044276,"remote_snapshot_timestamp":1726044333,"replay_state":"syncing","syncing_percent":0,"syncing_snapshot_timestamp":1726044333}42134269mirror.1smithi0772024-09-11 08:45:36cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:45:28' 2024-09-11T08:45:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:45:53.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:45:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:45:36' 2024-09-11T08:45:53.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:45:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:45:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:45:53.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:45:53.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:45:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:45:53.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap6 2024-09-11T08:45:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap6 2024-09-11T08:45:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:45:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:45:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:45:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap6 2024-09-11T08:45:53.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:46:05.225 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5645407148895325) > weight (0.25) 2024-09-11T08:46:05.225 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9913946137892583) > weight (0.25) 2024-09-11T08:46:05.225 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T08:46:05.226 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T08:46:05.226 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-11T08:46:05.227 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T08:46:05.226+0000 7f73b5a46640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 30821) UID: 1000 2024-09-11T08:46:12.227 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T08:46:12.227 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:46:12.228 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T08:46:12.228 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T08:46:12.228 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T08:46:12.232 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T08:46:12.232 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-09-11T08:46:22.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:22.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:46:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:46:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:46:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:46:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:46:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:46:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:46:23.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:46:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:46:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:46:23.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:46:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:46:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:46:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:46:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:46:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:46:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:46:23.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:46:23.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:46:23.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:46:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:46:23.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:46:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:46:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:46:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:46:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:46:23.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:46:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:46:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:46:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:46:23.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:46:23.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:46:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:46:23.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:46:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:46:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:46:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:46:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:46:23.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:46:23.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:46:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:46:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:46:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:46:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:46:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:46:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:46:23.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:46:23.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:46:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:46:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:46:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:46:23.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:46:23.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:46:23.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:46:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:46:23.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:46:23.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:23.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:46:23.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:46:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:46:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:46:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:46:23.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:46:23.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:46:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:46:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:23.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:46:25.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 20 2024-09-11T08:46:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:46:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:46:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:46:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:46:25.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:46:25.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:46:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=20 2024-09-11T08:46:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:46:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:46:25.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:46:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:25.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:46:25.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:46:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:46:26.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:46:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:46:26.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:46:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:46:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:46:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 20 = '' 2024-09-11T08:46:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:46:26.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:46:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:26.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:46:26.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:46:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:46:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:46:26.519 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:46:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:46:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:46:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:46:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 20 = '' 2024-09-11T08:46:26.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:46:26.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:46:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:27.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:46:27.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:46:27.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:46:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:46:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:46:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:46:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:46:27.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 20 = '' 2024-09-11T08:46:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:46:27.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:46:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:29.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:46:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:46:29.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:46:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:46:29.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=20 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 20 = 20 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap6 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap6 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:46:29.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:46:30.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap6 2024-09-11T08:46:30.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:46:30.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 12 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:45:43 2024 2024-09-11T08:46:30.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:46:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:46:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:46:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:46:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:46:30.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:46:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap5-clean ']' 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap5-clean 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:30.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:46:30.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:46:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:32.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:46:32.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:32.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap6 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap6 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:46:32.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:32.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:46:33.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:46:33.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:34.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:46:34.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:34.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:46:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap6-clean 2024-09-11T08:46:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap6-clean 2024-09-11T08:46:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:46:34.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap6-clean 2024-09-11T08:46:34.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap6-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:46:35.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:46:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:46:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:46:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:46:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:46:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:46:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:46:35.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 23306240 bytes 2024-09-11T08:46:36.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 43857920 bytes 2024-09-11T08:46:36.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 74465280 bytes 2024-09-11T08:46:36.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 90127360 bytes 2024-09-11T08:46:36.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 90738688 bytes 2024-09-11T08:46:36.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 116902912 bytes 2024-09-11T08:46:36.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 136684544 bytes 2024-09-11T08:46:36.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 193393664 bytes 2024-09-11T08:46:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 249846784 bytes 2024-09-11T08:46:37.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 289002496 bytes 2024-09-11T08:46:37.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 349095936 bytes 2024-09-11T08:46:37.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 468069376 bytes 2024-09-11T08:46:37.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 511833088 bytes 2024-09-11T08:46:37.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 561091584 bytes 2024-09-11T08:46:38.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 584235008 bytes 2024-09-11T08:46:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 628753408 bytes 2024-09-11T08:46:38.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 666287104 bytes 2024-09-11T08:46:38.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 679904256 bytes 2024-09-11T08:46:38.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 726102016 bytes 2024-09-11T08:46:38.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 774507520 bytes 2024-09-11T08:46:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 783579136 bytes 2024-09-11T08:46:39.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 843725824 bytes 2024-09-11T08:46:39.265 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 872975360 bytes 2024-09-11T08:46:39.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 910970880 bytes 2024-09-11T08:46:39.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 950051840 bytes 2024-09-11T08:46:39.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 972563456 bytes 2024-09-11T08:46:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 987478016 bytes 2024-09-11T08:46:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1045286912 bytes 2024-09-11T08:46:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1089532928 bytes 2024-09-11T08:46:40.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1150803968 bytes 2024-09-11T08:46:40.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1204916224 bytes 2024-09-11T08:46:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1271369728 bytes 2024-09-11T08:46:40.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1321161728 bytes 2024-09-11T08:46:41.121 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1359150080 bytes 2024-09-11T08:46:41.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1390231552 bytes 2024-09-11T08:46:41.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1451675648 bytes 2024-09-11T08:46:41.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1525687296 bytes 2024-09-11T08:46:41.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1627850752 bytes 2024-09-11T08:46:42.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1645534208 bytes 2024-09-11T08:46:42.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1699787776 bytes 2024-09-11T08:46:42.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1751422976 bytes 2024-09-11T08:46:42.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1798113280 bytes 2024-09-11T08:46:42.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1798727680 bytes 2024-09-11T08:46:42.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:46:42.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:46:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:46:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:43.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:46:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:46:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:46:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:46:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:46:43.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:46:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:46:43.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:46:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:46:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":17721890.13,"bytes_per_snapshot":533642752.0,"last_snapshot_bytes":531656704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044384,"remote_snapshot_timestamp":1726044384,"replay_state":"idle"} 2024-09-11T08:46:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:46:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:46:36 2024-09-11T08:46:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:46:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:46:43.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:46:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:46:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:46:28' 2024-09-11T08:46:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:46:43.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:46:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:46:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":17721890.13,"bytes_per_snapshot":533642752.0,"last_snapshot_bytes":531656704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044384,"remote_snapshot_timestamp":1726044384,"replay_state":"idle"} 2024-09-11T08:46:43.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:46:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:46:43.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:46:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:46:43.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:46:43.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:46:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:46:43.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:46:36' 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:46:36' 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:46:36' 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:46:43.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:46:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK43974400mirror.0smithi07719.1.1-298-gb53b6d3efalseOK44934495mirror.2smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":17721890.13,"bytes_per_snapshot":533642752.0,"last_snapshot_bytes":531656704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044384,"remote_snapshot_timestamp":1726044384,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:46:36cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:46:28' 2024-09-11T08:46:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:46:43.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:46:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:46:36' 2024-09-11T08:46:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:46:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:46:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:46:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:46:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:46:43.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap7 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap7 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap7 2024-09-11T08:46:43.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:46:53.234 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.2606213756349025) > weight (0.25) 2024-09-11T08:46:53.235 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.4780550569859384) > weight (0.25) 2024-09-11T08:46:53.235 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.47862745160552667) > weight (0.25) 2024-09-11T08:46:53.235 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4394631906309314) > weight (0.25) 2024-09-11T08:46:53.235 INFO:tasks.rbd_mirror_thrash:waiting for 90 secs before thrashing 2024-09-11T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:12.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:47:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:47:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:47:12.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:47:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:47:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:13.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:47:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:47:13.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:47:13.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:47:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:47:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:47:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:47:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:47:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:47:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:47:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:47:13.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:47:13.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:47:13.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:47:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:47:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:47:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:47:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:47:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:47:13.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:47:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:47:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:47:13.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:47:13.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:47:13.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:47:13.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:47:13.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:47:13.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:47:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:47:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:47:13.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:47:13.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:47:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:47:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:47:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:47:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:47:13.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:47:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:47:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:47:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:47:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:47:13.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:47:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:13.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:47:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:47:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:13.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:47:15.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 23 2024-09-11T08:47:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:47:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:47:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:15.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:47:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:47:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:47:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:47:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=23 2024-09-11T08:47:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:47:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:47:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:47:15.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:47:15.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:47:15.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:47:15.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:47:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:47:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:47:15.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:47:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:47:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 23 = '' 2024-09-11T08:47:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:47:15.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:47:16.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:47:16.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:47:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:47:16.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:47:16.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:47:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:47:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:47:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:47:16.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 23 = '' 2024-09-11T08:47:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:47:16.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:47:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:17.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:47:17.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:47:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:47:17.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:47:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:47:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:47:17.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:47:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:47:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 23 = '' 2024-09-11T08:47:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:47:17.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:47:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:47:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:47:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:47:18.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:47:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=23 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 23 = 23 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap7 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap7 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:47:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:47:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap7 2024-09-11T08:47:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:47:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:47:20.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 14 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:45:43 2024 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:47:20.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:47:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:47:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap6-clean ']' 2024-09-11T08:47:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-11T08:47:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap6-clean 2024-09-11T08:47:20.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:47:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:47:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:47:20.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:47:20.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:21.704 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:47:21.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap7 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap7 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:47:22.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:22.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:47:22.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:47:22.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:23.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:47:23.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:24.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:47:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap7-clean 2024-09-11T08:47:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap7-clean 2024-09-11T08:47:24.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:47:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap7-clean 2024-09-11T08:47:24.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap7-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:47:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:47:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:47:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:47:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:24.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:24.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:47:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:47:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:47:24.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:47:24.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 48497664 bytes 2024-09-11T08:47:25.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 89640960 bytes 2024-09-11T08:47:25.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 127843328 bytes 2024-09-11T08:47:25.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 166431744 bytes 2024-09-11T08:47:25.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 178041856 bytes 2024-09-11T08:47:25.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 225775616 bytes 2024-09-11T08:47:25.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 239681536 bytes 2024-09-11T08:47:25.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 341603328 bytes 2024-09-11T08:47:26.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 386971648 bytes 2024-09-11T08:47:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 458299392 bytes 2024-09-11T08:47:26.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 468536320 bytes 2024-09-11T08:47:26.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 524484608 bytes 2024-09-11T08:47:26.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 548211712 bytes 2024-09-11T08:47:26.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 611759104 bytes 2024-09-11T08:47:27.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 623453184 bytes 2024-09-11T08:47:27.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 657166336 bytes 2024-09-11T08:47:27.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 669906944 bytes 2024-09-11T08:47:27.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 684366848 bytes 2024-09-11T08:47:27.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 709046272 bytes 2024-09-11T08:47:27.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 752059392 bytes 2024-09-11T08:47:27.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 799583232 bytes 2024-09-11T08:47:28.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 859102208 bytes 2024-09-11T08:47:28.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 919929856 bytes 2024-09-11T08:47:28.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 938188800 bytes 2024-09-11T08:47:28.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 938790912 bytes 2024-09-11T08:47:28.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 939410432 bytes 2024-09-11T08:47:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 969612288 bytes 2024-09-11T08:47:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1007684608 bytes 2024-09-11T08:47:29.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1058571264 bytes 2024-09-11T08:47:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1091798016 bytes 2024-09-11T08:47:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1104361472 bytes 2024-09-11T08:47:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1127572480 bytes 2024-09-11T08:47:29.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1169352704 bytes 2024-09-11T08:47:29.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1195113472 bytes 2024-09-11T08:47:29.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1219106816 bytes 2024-09-11T08:47:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1219717120 bytes 2024-09-11T08:47:29.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1253538816 bytes 2024-09-11T08:47:30.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1347006464 bytes 2024-09-11T08:47:30.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1380672512 bytes 2024-09-11T08:47:30.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1401530368 bytes 2024-09-11T08:47:30.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1470478336 bytes 2024-09-11T08:47:30.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1550513152 bytes 2024-09-11T08:47:31.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1599063040 bytes 2024-09-11T08:47:31.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1660803072 bytes 2024-09-11T08:47:31.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1680880640 bytes 2024-09-11T08:47:31.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1701482496 bytes 2024-09-11T08:47:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:47:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:47:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:47:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:31.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:47:31.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:47:32.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:47:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:47:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:47:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:47:32.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:47:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:47:32.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:47:32.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533642752.0,"last_snapshot_bytes":531656704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044384,"remote_snapshot_timestamp":1726044384,"replay_state":"idle"} 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:47:06 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:47:32.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:47:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:47:28' 2024-09-11T08:47:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:47:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:47:32.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:47:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533642752.0,"last_snapshot_bytes":531656704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044384,"remote_snapshot_timestamp":1726044384,"replay_state":"idle"} 2024-09-11T08:47:32.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:47:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:47:32.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:47:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:47:32.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:47:32.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:47:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:47:32.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:47:32.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:47:06' 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:47:06' 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:47:06' 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:47:32.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:47:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK43974400mirror.0smithi07719.1.1-298-gb53b6d3efalseOK44934495mirror.2smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":533642752.0,"last_snapshot_bytes":531656704,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044384,"remote_snapshot_timestamp":1726044384,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:47:06cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:47:28' 2024-09-11T08:47:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:47:32.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:47:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:47:06' 2024-09-11T08:47:32.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:47:32.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:47:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:47:32.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:47:32.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:47:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:47:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap8 2024-09-11T08:47:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap8 2024-09-11T08:47:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:47:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:47:32.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:47:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap8 2024-09-11T08:47:32.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:47:50.233 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:47:50.234 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:47:50.234 INFO:tasks.rbd_mirror_thrash:waiting for 9 secs before reviving daemons 2024-09-11T08:47:50.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:47:50.237+0000 7f8ac2567640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 33428) UID: 1000 2024-09-11T08:47:59.234 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:47:59.235 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:47:59.235 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:47:59.235 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:47:59.235 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:47:59.239 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:47:59.239 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-11T08:48:02.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:48:02.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:48:02.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:48:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:48:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:03.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:48:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:48:03.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:48:03.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:48:03.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:48:03.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:48:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:48:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:48:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:48:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:48:03.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:48:03.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:48:03.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:48:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:48:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:48:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:48:03.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:48:03.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:48:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:48:03.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:48:03.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:48:03.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:03.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:48:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:48:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:48:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:48:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:48:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:48:03.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:48:03.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:48:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:48:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:48:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:48:03.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:48:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:48:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:48:03.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:48:03.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:48:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:48:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:48:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:48:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:48:03.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:03.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:48:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:48:03.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:48:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:48:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:48:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:48:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:48:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 26 2024-09-11T08:48:05.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:48:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:48:05.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:48:05.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:48:05.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:48:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=26 2024-09-11T08:48:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:48:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:48:05.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:48:05.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:48:05.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:48:06.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:48:06.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:48:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:48:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:48:06.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 26 = '' 2024-09-11T08:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:48:06.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:48:06.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:06.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:48:06.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:48:06.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:48:06.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:48:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:48:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:48:06.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:48:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:48:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 26 = '' 2024-09-11T08:48:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:48:06.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:48:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:48:07.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:48:07.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:48:07.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:48:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:48:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:48:07.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:48:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:48:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 26 = '' 2024-09-11T08:48:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:48:07.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:48:09.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:48:09.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:48:09.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:48:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=26 2024-09-11T08:48:09.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 26 = 26 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap8 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap8 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:48:09.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:48:10.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap8 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 16 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:48:10.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:48:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:47:20 2024 2024-09-11T08:48:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:48:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:48:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:48:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:48:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:48:10.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap7-clean ']' 2024-09-11T08:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-11T08:48:10.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap7-clean 2024-09-11T08:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:10.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:48:11.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:48:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:12.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:48:12.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:12.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap8 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap8 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:12.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:12.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:48:13.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:48:13.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:14.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:48:14.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap8-clean 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap8-clean 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap8-clean 2024-09-11T08:48:14.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap8-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:48:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:48:15.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:48:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:15.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:48:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=11 2024-09-11T08:48:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:48:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:48:16.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 125701120 bytes 2024-09-11T08:48:16.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 173922304 bytes 2024-09-11T08:48:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 251482112 bytes 2024-09-11T08:48:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 297099264 bytes 2024-09-11T08:48:16.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 369618944 bytes 2024-09-11T08:48:17.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 412481536 bytes 2024-09-11T08:48:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 471075840 bytes 2024-09-11T08:48:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 502424576 bytes 2024-09-11T08:48:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 530785280 bytes 2024-09-11T08:48:17.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 613649408 bytes 2024-09-11T08:48:18.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 691969024 bytes 2024-09-11T08:48:18.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 749375488 bytes 2024-09-11T08:48:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 764966912 bytes 2024-09-11T08:48:18.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 833047552 bytes 2024-09-11T08:48:18.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 876704768 bytes 2024-09-11T08:48:18.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 908695552 bytes 2024-09-11T08:48:19.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 962434048 bytes 2024-09-11T08:48:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 1017982976 bytes 2024-09-11T08:48:19.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1121737728 bytes 2024-09-11T08:48:19.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1182628864 bytes 2024-09-11T08:48:20.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1254708224 bytes 2024-09-11T08:48:20.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1317164032 bytes 2024-09-11T08:48:20.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1363508224 bytes 2024-09-11T08:48:20.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1397373952 bytes 2024-09-11T08:48:20.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1433138176 bytes 2024-09-11T08:48:20.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1461018624 bytes 2024-09-11T08:48:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1486091264 bytes 2024-09-11T08:48:21.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1503321088 bytes 2024-09-11T08:48:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1512725504 bytes 2024-09-11T08:48:21.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1614653440 bytes 2024-09-11T08:48:21.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1684043776 bytes 2024-09-11T08:48:21.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1722501120 bytes 2024-09-11T08:48:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1762176000 bytes 2024-09-11T08:48:22.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1805127680 bytes 2024-09-11T08:48:22.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1823711232 bytes 2024-09-11T08:48:22.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1856534528 bytes 2024-09-11T08:48:22.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1860111360 bytes 2024-09-11T08:48:22.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1873902592 bytes 2024-09-11T08:48:22.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1980423168 bytes 2024-09-11T08:48:23.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 2012073984 bytes 2024-09-11T08:48:23.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 2087229440 bytes 2024-09-11T08:48:23.236 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T08:48:23.236 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T08:48:23.237 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-11T08:48:23.237 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T08:48:23.236+0000 7fe357dcd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 33520) UID: 1000 2024-09-11T08:48:23.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 2153798656 bytes 2024-09-11T08:48:23.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 2193528832 bytes 2024-09-11T08:48:23.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 2228278272 bytes 2024-09-11T08:48:23.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2244303872 bytes 2024-09-11T08:48:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2332068864 bytes 2024-09-11T08:48:24.238 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T08:48:24.238 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:48:24.238 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T08:48:24.238 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T08:48:24.238 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T08:48:24.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2344071168 bytes 2024-09-11T08:48:24.241 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T08:48:24.241 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-11T08:48:24.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2386140160 bytes 2024-09-11T08:48:24.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2450738176 bytes 2024-09-11T08:48:24.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2483781632 bytes 2024-09-11T08:48:24.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2522855424 bytes 2024-09-11T08:48:25.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2555931648 bytes 2024-09-11T08:48:25.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2605199360 bytes 2024-09-11T08:48:25.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2618948608 bytes 2024-09-11T08:48:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2689785856 bytes 2024-09-11T08:48:25.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2719839232 bytes 2024-09-11T08:48:25.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2764628992 bytes 2024-09-11T08:48:26.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2804188160 bytes 2024-09-11T08:48:26.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2808131584 bytes 2024-09-11T08:48:26.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2872467456 bytes 2024-09-11T08:48:26.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2908118016 bytes 2024-09-11T08:48:26.514 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2908705792 bytes 2024-09-11T08:48:26.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2949859328 bytes 2024-09-11T08:48:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:48:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:48:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:48:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:48:26.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:48:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:48:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:48:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:48:26.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:48:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:48:27.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:48:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:48:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:48:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:48:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:48:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":522633728.0,"last_snapshot_bytes":513610752,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044434,"remote_snapshot_timestamp":1726044434,"replay_state":"idle"} 2024-09-11T08:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:48:27.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:48:06 2024-09-11T08:48:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:48:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:48:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:48:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:48:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:47:58' 2024-09-11T08:48:27.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:48:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:48:27.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:48:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":0.0,"bytes_per_snapshot":522633728.0,"last_snapshot_bytes":513610752,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044434,"remote_snapshot_timestamp":1726044434,"replay_state":"idle"} 2024-09-11T08:48:27.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:48:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:48:27.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:48:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:48:27.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:48:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:48:27.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:48:27.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:48:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:48:06' 2024-09-11T08:48:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:48:06' 2024-09-11T08:48:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:48:27.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:48:06' 2024-09-11T08:48:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:48:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:48:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:48:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK44934495mirror.2smithi07719.1.1-298-gb53b6d3efalseOK45554558mirror.0smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":0.0,"bytes_per_snapshot":522633728.0,"last_snapshot_bytes":513610752,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044434,"remote_snapshot_timestamp":1726044434,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:48:06cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:47:58' 2024-09-11T08:48:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:48:27.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:48:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:48:06' 2024-09-11T08:48:27.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:48:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:48:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:48:27.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:48:27.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap9 2024-09-11T08:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap9 2024-09-11T08:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:27.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap9 2024-09-11T08:48:27.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:48:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:48:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:48:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:48:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:48:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:48:58.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:48:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:48:58.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:48:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:48:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:48:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:48:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:48:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:48:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:48:58.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:48:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:48:58.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:48:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:48:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:48:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:48:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:48:58.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:48:58.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:48:58.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:48:58.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:48:58.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:48:58.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:48:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:48:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:48:58.376 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:48:58.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:48:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:48:58.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:48:58.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:48:58.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:48:58.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:48:58.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:48:58.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:48:58.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:48:58.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:48:58.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:48:58.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:48:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:48:58.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:58.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:48:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:48:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:48:58.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:48:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:48:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:48:58.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:48:58.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:49:00.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 29 2024-09-11T08:49:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:49:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:49:00.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:49:00.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:49:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=29 2024-09-11T08:49:00.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:00.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:00.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:49:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:49:00.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:49:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:49:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 29 = '' 2024-09-11T08:49:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:00.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:49:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:01.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:01.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:49:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:49:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:49:01.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:49:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 29 = '' 2024-09-11T08:49:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:01.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:49:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:02.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:02.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:02.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:49:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:49:02.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:49:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:49:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 29 = '' 2024-09-11T08:49:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:02.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:49:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:03.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:03.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:03.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:03.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=29 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 29 = 29 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap9 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap9 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:49:03.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:49:04.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap9 2024-09-11T08:49:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:49:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:49:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 18 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:47:20 2024 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:49:04.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:49:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:49:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:49:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:49:04.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:49:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap8-clean ']' 2024-09-11T08:49:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-11T08:49:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap8-clean 2024-09-11T08:49:04.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:49:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:04.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:05.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:05.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:06.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:06.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap9 2024-09-11T08:49:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap9 2024-09-11T08:49:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:49:07.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:07.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:07.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:08.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:08.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:09.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap9-clean 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap9-clean 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap9-clean 2024-09-11T08:49:09.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap9-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:49:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:49:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror test 2024-09-11T08:49:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:09.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:09.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:49:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:49:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:49:09.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror test --debug-rbd=20 --debug-journaler=20 2024-09-11T08:49:09.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 48464896 bytes 2024-09-11T08:49:10.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 141552640 bytes 2024-09-11T08:49:10.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 176673792 bytes 2024-09-11T08:49:10.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 294851584 bytes 2024-09-11T08:49:10.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 347245568 bytes 2024-09-11T08:49:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 380335104 bytes 2024-09-11T08:49:11.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 459080704 bytes 2024-09-11T08:49:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 494839808 bytes 2024-09-11T08:49:11.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 495440896 bytes 2024-09-11T08:49:11.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 544818176 bytes 2024-09-11T08:49:11.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 545443840 bytes 2024-09-11T08:49:11.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 606717952 bytes 2024-09-11T08:49:11.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 671699968 bytes 2024-09-11T08:49:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 724980736 bytes 2024-09-11T08:49:12.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 745441280 bytes 2024-09-11T08:49:12.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 783280128 bytes 2024-09-11T08:49:12.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 851472384 bytes 2024-09-11T08:49:12.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 934053888 bytes 2024-09-11T08:49:13.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 957742080 bytes 2024-09-11T08:49:13.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1111265280 bytes 2024-09-11T08:49:13.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1131609088 bytes 2024-09-11T08:49:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1173603328 bytes 2024-09-11T08:49:13.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1229999104 bytes 2024-09-11T08:49:14.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1278642176 bytes 2024-09-11T08:49:14.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1305080832 bytes 2024-09-11T08:49:14.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1348892672 bytes 2024-09-11T08:49:14.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1370265600 bytes 2024-09-11T08:49:14.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1403692032 bytes 2024-09-11T08:49:14.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1419342848 bytes 2024-09-11T08:49:14.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1441323008 bytes 2024-09-11T08:49:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1533410304 bytes 2024-09-11T08:49:15.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1588216832 bytes 2024-09-11T08:49:15.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1619955712 bytes 2024-09-11T08:49:15.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1671848960 bytes 2024-09-11T08:49:15.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1691841536 bytes 2024-09-11T08:49:15.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1727039488 bytes 2024-09-11T08:49:16.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1764884480 bytes 2024-09-11T08:49:16.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1828731904 bytes 2024-09-11T08:49:16.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1845511168 bytes 2024-09-11T08:49:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:49:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:49:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:49:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_status_in_pool_dir cluster1 mirror test up+replaying 2024-09-11T08:49:16.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:49:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:49:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:49:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:49:16.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:49:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_status_in_pool_dir cluster1 mirror test up+replaying '' '' 2024-09-11T08:49:17.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local description_pattern= 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local service_pattern= 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:49:17.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror image status mirror/test 2024-09-11T08:49:17.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='test: 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: global_id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+replaying 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: replaying, {"bytes_per_second":17658094.93,"bytes_per_snapshot":521948160.0,"last_snapshot_bytes":529742848,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044539,"remote_snapshot_timestamp":1726044539,"replay_state":"idle"} 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: service: mirror.1 on smithi077 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:49:06 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: peer_sites: 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: name: cluster2 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: state: up+stopped 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: description: local image is primary 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: last_update: 2024-09-11 08:48:58' 2024-09-11T08:49:17.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ state: .*up+replaying' 2024-09-11T08:49:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: state: up+replaying 2024-09-11T08:49:17.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^ description: .*' 2024-09-11T08:49:17.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: description: replaying, {"bytes_per_second":17658094.93,"bytes_per_snapshot":521948160.0,"last_snapshot_bytes":529742848,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044539,"remote_snapshot_timestamp":1726044539,"replay_state":"idle"} 2024-09-11T08:49:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:49:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:49:17.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep '^up+' 2024-09-11T08:49:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:49:17.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep 'service: *mirror..* on ' 2024-09-11T08:49:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: service: mirror.1 on smithi077 2024-09-11T08:49:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update 2024-09-11T08:49:17.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -nEe 's/^ last_update: *(.*) *$/\1/p' 2024-09-11T08:49:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:49:06' 2024-09-11T08:49:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_mirror_pool_status_verbose cluster1 mirror test up+replaying '2024-09-11 08:49:06' 2024-09-11T08:49:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:17.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state_pattern=up+replaying 2024-09-11T08:49:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local 'prev_last_update=2024-09-11 08:49:06' 2024-09-11T08:49:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status 2024-09-11T08:49:17.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ CEPH_ARGS= 2024-09-11T08:49:17.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 mirror pool status mirror --verbose --format xml 2024-09-11T08:49:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status='OKOKOK142134269mirror.1smithi07719.1.1-298-gb53b6d3etrueOK42214275mirror.3smithi07719.1.1-298-gb53b6d3efalseOK44934495mirror.2smithi07719.1.1-298-gb53b6d3efalseOK45554558mirror.0smithi07719.1.1-298-gb53b6d3efalseOKtesta2599b3a-7c6a-4d17-ac98-f264ae7b005cup+replayingreplaying, {"bytes_per_second":17658094.93,"bytes_per_snapshot":521948160.0,"last_snapshot_bytes":529742848,"last_snapshot_sync_seconds":1,"local_snapshot_timestamp":1726044539,"remote_snapshot_timestamp":1726044539,"replay_state":"idle"}42134269mirror.1smithi0772024-09-11 08:49:06cluster216cd46a8-7a95-494f-a9ad-bfcebd7166e0up+stoppedlocal image is primary2024-09-11 08:48:58' 2024-09-11T08:49:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_update state 2024-09-11T08:49:17.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/last_update' 2024-09-11T08:49:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ last_update='2024-09-11 08:49:06' 2024-09-11T08:49:17.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v '//images/image[name='\''test'\'']/state' 2024-09-11T08:49:17.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=up+replaying 2024-09-11T08:49:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo up+replaying 2024-09-11T08:49:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep up+replaying 2024-09-11T08:49:17.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:up+replaying 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap10 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap10 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap10 2024-09-11T08:49:17.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:49:42.240 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:49:42.241 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:49:42.241 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T08:49:42.242 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:49:42.243+0000 7fef07baa640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 35680) UID: 1000 2024-09-11T08:49:45.242 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:49:45.242 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:49:45.243 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:49:45.243 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:49:45.243 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:49:45.246 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:49:45.247 INFO:tasks.rbd_mirror_thrash:waiting for 91 secs before thrashing 2024-09-11T08:49:47.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:49:47.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:49:48.242 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7442646973096607) > weight (0.25) 2024-09-11T08:49:48.242 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T08:49:48.243 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T08:49:48.243 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T08:49:48.244 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T08:49:48.247+0000 7ff1d6b03640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 30882) UID: 1000 2024-09-11T08:49:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:49:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:48.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:49:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:49:48.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:49:48.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:49:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:49:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:49:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:49:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:49:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:49:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:49:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:49:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:49:48.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:49:48.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:49:48.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:49:48.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:49:48.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:49:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:49:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:49:48.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:49:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:49:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:49:48.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:49:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:49:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:49:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:49:48.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:49:48.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:49:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:49:48.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:49:48.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:49:48.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:49:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:49:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:49:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:49:48.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:49:48.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:49:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:49:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:49:48.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:49:48.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:49:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:49:48.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:49:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:49:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:49:48.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:49:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:49:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:48.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:49:50.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 32 2024-09-11T08:49:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:49:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:50.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:49:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:49:50.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:50.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:49:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=32 2024-09-11T08:49:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:50.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:50.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:50.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:51.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:49:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:49:51.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:49:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:49:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 32 = '' 2024-09-11T08:49:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:51.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T08:49:51.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:51.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:51.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:49:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:49:51.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:49:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:49:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 32 = '' 2024-09-11T08:49:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:51.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T08:49:52.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:52.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:52.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:52.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:49:52.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T08:49:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T08:49:52.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T08:49:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T08:49:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 32 = '' 2024-09-11T08:49:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:49:52.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T08:49:54.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:49:54.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:49:54.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:49:54.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:49:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=32 2024-09-11T08:49:54.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 32 = 32 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap10 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap10 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:49:54.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:49:54.244 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T08:49:54.244 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:49:54.245 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T08:49:54.245 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T08:49:54.245 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T08:49:54.248 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T08:49:54.248 INFO:tasks.rbd_mirror_thrash:waiting for 83 secs before thrashing 2024-09-11T08:49:55.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap10 2024-09-11T08:49:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:49:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:49:55.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 20 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:49:55.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:49:05 2024 2024-09-11T08:49:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:49:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:49:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:49:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:49:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:49:55.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:49:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snap9-clean ']' 2024-09-11T08:49:55.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap9-clean 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:55.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:56.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:57.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap10 2024-09-11T08:49:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap10 2024-09-11T08:49:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:49:57.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:57.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:49:58.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:58.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:59.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:49:59.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:59.487 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:49:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:49:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ clean_snap_name=snap10-clean 2024-09-11T08:49:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror test snap10-clean 2024-09-11T08:49:59.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:49:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:49:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:49:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap10-clean 2024-09-11T08:49:59.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create test@snap10-clean --debug-rbd=20 --debug-journaler=20 2024-09-11T08:50:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror test 2024-09-11T08:50:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:50:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:00.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror test started 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:50:00.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:50:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror test started 2024-09-11T08:50:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:50:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:50:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:50:01.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:50:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/test 2024-09-11T08:50:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:50:01.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:50:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:50:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:50:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:50:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:50:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:50:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:50:01.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:50:01.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/test 2024-09-11T08:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:50:01.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:50:01.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:50:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:50:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:50:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:50:01.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:50:01.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/test 2024-09-11T08:50:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/test'] 2024-09-11T08:50:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T08:50:01.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T08:50:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:50:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/test 2024-09-11T08:50:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T08:50:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:50:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:50:01.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T08:50:01.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T08:50:01.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:50:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:50:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T08:50:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T08:50:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T08:50:01.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:50:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:50:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T08:50:01.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/test 2024-09-11T08:50:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:50:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:50:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:50:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:50:01.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:50:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:50:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:50:01.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror test 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror test 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:01.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-test.status 2024-09-11T08:50:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-test.status 2024-09-11T08:50:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:50:01.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:50:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-test.status 2024-09-11T08:50:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-test.status 2024-09-11T08:50:01.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:50:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:50:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror test 2024-09-11T08:50:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:01.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/test 2024-09-11T08:50:03.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 34 2024-09-11T08:50:03.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror test /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:50:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.status 2024-09-11T08:50:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/test --format xml 2024-09-11T08:50:03.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:50:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=34 2024-09-11T08:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:50:03.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T08:50:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror test /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:50:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:50:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.status 2024-09-11T08:50:03.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/test --format xml 2024-09-11T08:50:03.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:50:03.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=34 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 34 = 34 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror test snap10-clean 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap10-clean 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T08:50:03.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:50:04.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/test@snap10-clean 2024-09-11T08:50:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'test': 2024-09-11T08:50:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T08:50:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:50:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 21 2024-09-11T08:50:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10aa4921f287 2024-09-11T08:50:05.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10aa4921f287 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:49:05 2024 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:41:08 2024 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a2599b3a-7c6a-4d17-ac98-f264ae7b005c 2024-09-11T08:50:05.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T08:50:05.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:05.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 10 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap1 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap1 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap1 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:05.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:05.259 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:05.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:05.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:07.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:07.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:07.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap1-clean 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap1-clean 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap1-clean 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:07.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:07.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap1-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:08.407 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:08.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap1-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:09.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:09.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:09.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap2 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap2 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap2 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:09.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:10.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:10.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:11.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:11.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap2-clean 2024-09-11T08:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap2-clean 2024-09-11T08:50:12.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap2-clean 2024-09-11T08:50:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:12.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap2-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:13.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:13.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap2-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:14.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:14.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap3 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap3 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap3 2024-09-11T08:50:14.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:14.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:15.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:15.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:16.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:16.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:16.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap3-clean 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap3-clean 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap3-clean 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:17.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:17.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap3-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:17.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:17.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap3-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:18.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:18.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:19.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:19.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap4 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap4 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap4 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:19.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:21.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:21.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap4-clean 2024-09-11T08:50:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap4-clean 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap4-clean 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap4-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:22.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap4-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:23.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:23.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:23.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap5 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap5 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap5 2024-09-11T08:50:24.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:24.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:24.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:24.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:25.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:25.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:26.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap5-clean 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap5-clean 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap5-clean 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:26.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap5-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:26.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:26.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap5-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:28.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:28.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:28.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:28.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap6 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap6 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap6 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:28.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:29.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:29.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:30.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:30.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:30.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap6-clean 2024-09-11T08:50:30.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap6-clean 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap6-clean 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:30.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap6-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:31.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap6-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:32.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:32.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:32.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap7 2024-09-11T08:50:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap7 2024-09-11T08:50:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:33.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap7 2024-09-11T08:50:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:33.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:33.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:33.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:34.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:34.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:35.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap7-clean 2024-09-11T08:50:35.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap7-clean 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap7-clean 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:35.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap7-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:35.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:35.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap7-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:37.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:37.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:37.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap8 2024-09-11T08:50:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap8 2024-09-11T08:50:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap8 2024-09-11T08:50:37.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:37.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:38.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:38.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:39.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:39.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap8-clean 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap8-clean 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap8-clean 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:39.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:39.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap8-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:40.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:40.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap8-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:41.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:41.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:42.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap9 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap9 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap9 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:42.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:42.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:42.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:42.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:43.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:44.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap9-clean 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap9-clean 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap9-clean 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:44.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:44.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap9-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:45.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap9-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:46.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:46.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:46.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap10 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap10 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap10 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:46.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:47.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:47.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:48.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:48.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:48.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:48.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap10-clean 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror test snap10-clean 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap10-clean 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:48.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:48.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export test@snap10-clean /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export 2024-09-11T08:50:49.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:49.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export test@snap10-clean /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:50.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 1% complete... Exporting image: 2% complete... Exporting image: 3% complete... Exporting image: 4% complete... Exporting image: 5% complete... Exporting image: 6% complete... Exporting image: 7% complete... Exporting image: 8% complete... Exporting image: 9% complete... Exporting image: 10% complete... Exporting image: 11% complete... Exporting image: 12% complete... Exporting image: 13% complete... Exporting image: 14% complete... Exporting image: 15% complete... Exporting image: 16% complete... Exporting image: 17% complete... Exporting image: 18% complete... Exporting image: 19% complete... Exporting image: 20% complete... Exporting image: 21% complete... Exporting image: 22% complete... Exporting image: 23% complete... Exporting image: 24% complete... Exporting image: 25% complete... Exporting image: 26% complete... Exporting image: 27% complete... Exporting image: 28% complete... Exporting image: 29% complete... Exporting image: 30% complete... Exporting image: 31% complete... Exporting image: 32% complete... Exporting image: 33% complete... Exporting image: 34% complete... Exporting image: 35% complete... Exporting image: 36% complete... Exporting image: 37% complete... Exporting image: 38% complete... Exporting image: 39% complete... Exporting image: 40% complete... Exporting image: 41% complete... Exporting image: 42% complete... Exporting image: 43% complete... Exporting image: 44% complete... Exporting image: 45% complete... Exporting image: 46% complete... Exporting image: 47% complete... Exporting image: 48% complete... Exporting image: 49% complete... Exporting image: 50% complete... Exporting image: 51% complete... Exporting image: 52% complete... Exporting image: 53% complete... Exporting image: 54% complete... Exporting image: 55% complete... Exporting image: 56% complete... Exporting image: 57% complete... Exporting image: 58% complete... Exporting image: 59% complete... Exporting image: 60% complete... Exporting image: 61% complete... Exporting image: 62% complete... Exporting image: 63% complete... Exporting image: 64% complete... Exporting image: 65% complete... Exporting image: 66% complete... Exporting image: 67% complete... Exporting image: 68% complete... Exporting image: 69% complete... Exporting image: 70% complete... Exporting image: 71% complete... Exporting image: 72% complete... Exporting image: 73% complete... Exporting image: 74% complete... Exporting image: 75% complete... Exporting image: 76% complete... Exporting image: 77% complete... Exporting image: 78% complete... Exporting image: 79% complete... Exporting image: 80% complete... Exporting image: 81% complete... Exporting image: 82% complete... Exporting image: 83% complete... Exporting image: 84% complete... Exporting image: 85% complete... Exporting image: 86% complete... Exporting image: 87% complete... Exporting image: 88% complete... Exporting image: 89% complete... Exporting image: 90% complete... Exporting image: 91% complete... Exporting image: 92% complete... Exporting image: 93% complete... Exporting image: 94% complete... Exporting image: 95% complete... Exporting image: 96% complete... Exporting image: 97% complete... Exporting image: 98% complete... Exporting image: 99% complete... Exporting image: 100% complete...done. 2024-09-11T08:50:50.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:50.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-test.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-test.export 2024-09-11T08:50:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:50:51.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 10 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap1 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap1 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap1 2024-09-11T08:50:51.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1 2024-09-11T08:50:51.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap1-clean 2024-09-11T08:50:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap1-clean 2024-09-11T08:50:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap1-clean 2024-09-11T08:50:51.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap1-clean 2024-09-11T08:50:52.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap2 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap2 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap2 2024-09-11T08:50:52.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2 2024-09-11T08:50:53.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap2-clean 2024-09-11T08:50:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap2-clean 2024-09-11T08:50:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap2-clean 2024-09-11T08:50:53.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap2-clean 2024-09-11T08:50:54.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap3 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap3 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap3 2024-09-11T08:50:54.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3 2024-09-11T08:50:55.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap3-clean 2024-09-11T08:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap3-clean 2024-09-11T08:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap3-clean 2024-09-11T08:50:55.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap3-clean 2024-09-11T08:50:56.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:56.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap4 2024-09-11T08:50:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap4 2024-09-11T08:50:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap4 2024-09-11T08:50:56.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4 2024-09-11T08:50:57.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap4-clean 2024-09-11T08:50:57.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap4-clean 2024-09-11T08:50:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap4-clean 2024-09-11T08:50:57.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap4-clean 2024-09-11T08:50:58.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap5 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap5 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap5 2024-09-11T08:50:58.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5 2024-09-11T08:50:59.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:50:59.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap5-clean 2024-09-11T08:50:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap5-clean 2024-09-11T08:50:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:50:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:50:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:50:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap5-clean 2024-09-11T08:50:59.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap5-clean 2024-09-11T08:51:00.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:51:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap6 2024-09-11T08:51:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap6 2024-09-11T08:51:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:00.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap6 2024-09-11T08:51:00.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6 2024-09-11T08:51:01.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:01.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap6-clean 2024-09-11T08:51:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap6-clean 2024-09-11T08:51:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap6-clean 2024-09-11T08:51:01.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap6-clean 2024-09-11T08:51:02.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:51:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap7 2024-09-11T08:51:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap7 2024-09-11T08:51:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:02.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap7 2024-09-11T08:51:02.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7 2024-09-11T08:51:03.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap7-clean 2024-09-11T08:51:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap7-clean 2024-09-11T08:51:03.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap7-clean 2024-09-11T08:51:03.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap7-clean 2024-09-11T08:51:04.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap8 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap8 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap8 2024-09-11T08:51:04.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8 2024-09-11T08:51:05.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap8-clean 2024-09-11T08:51:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap8-clean 2024-09-11T08:51:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap8-clean 2024-09-11T08:51:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap8-clean 2024-09-11T08:51:06.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:51:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap9 2024-09-11T08:51:06.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap9 2024-09-11T08:51:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap9 2024-09-11T08:51:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9 2024-09-11T08:51:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap9-clean 2024-09-11T08:51:07.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap9-clean 2024-09-11T08:51:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap9-clean 2024-09-11T08:51:07.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap9-clean 2024-09-11T08:51:08.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T08:51:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap10 2024-09-11T08:51:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap10 2024-09-11T08:51:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:08.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap10 2024-09-11T08:51:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10 2024-09-11T08:51:09.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap10-clean 2024-09-11T08:51:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror test snap10-clean 2024-09-11T08:51:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:09.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap10-clean 2024-09-11T08:51:09.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/test@snap10-clean 2024-09-11T08:51:10.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T08:51:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror test 2024-09-11T08:51:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T08:51:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T08:51:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror test 2024-09-11T08:51:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/test 2024-09-11T08:51:10.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/test 2024-09-11T08:51:10.956 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T08:51:10.957+0000 7fd66502e640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x562db3351d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T08:51:10.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T08:51:10.957+0000 7fd66b03a640 -1 rbd::mirror::ImageReplayer: 0x562db3ae0000 [3/a2599b3a-7c6a-4d17-ac98-f264ae7b005c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T08:51:12.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T08:51:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:12.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror test deleted 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror test 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T08:51:12.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=test 2024-09-11T08:51:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-11T08:51:12.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T08:51:12.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-11T08:51:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T08:51:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T08:51:12.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T08:51:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror test deleted '' 2024-09-11T08:51:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:51:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=test 2024-09-11T08:51:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T08:51:12.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T08:51:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T08:51:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T08:51:12.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror test 2024-09-11T08:51:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:51:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T08:51:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=test 2024-09-11T08:51:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/test 2024-09-11T08:51:12.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T08:51:12.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image test: (2) No such file or directory 2024-09-11T08:51:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T08:51:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T08:51:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T08:51:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ testlog 'TEST: create many images' 2024-09-11T08:51:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ tee -a /tmp/tmp.LFCa6yxq8J/rbd-mirror.test.log 2024-09-11T08:51:12.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ date '+%F %T' 2024-09-11T08:51:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo 2024-09-11 08:51:12 TEST: create many images 2024-09-11T08:51:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:2024-09-11 08:51:12 TEST: create many images 2024-09-11T08:51:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ snap_name=snap 2024-09-11T08:51:12.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 50 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_1 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_1 snapshot 128M 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:51:12.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_1 128M 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:51:12.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:51:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:12.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_1 2024-09-11T08:51:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:51:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:51:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:12.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_1 snapshot 2024-09-11T08:51:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:51:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:12.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-11T08:51:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:51:12.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-11T08:51:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_1': 2024-09-11T08:51:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:51:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:51:12.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1252c3ad5159 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1252c3ad5159 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:12 2024 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:12 2024 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:12 2024 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 209601d8-0ce8-4bab-bfd1-bea5e9278d31 2024-09-11T08:51:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:51:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:51:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-11T08:51:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:51:12.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:51:12.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T08:51:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:51:12.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:51:13.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 25732096 bytes 2024-09-11T08:51:13.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 96936960 bytes 2024-09-11T08:51:13.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 97515520 bytes 2024-09-11T08:51:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 190262272 bytes 2024-09-11T08:51:13.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 309136384 bytes 2024-09-11T08:51:14.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 335276032 bytes 2024-09-11T08:51:14.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 395810816 bytes 2024-09-11T08:51:14.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 406947840 bytes 2024-09-11T08:51:14.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 417933312 bytes 2024-09-11T08:51:14.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 436425728 bytes 2024-09-11T08:51:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 489968640 bytes 2024-09-11T08:51:15.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 584308736 bytes 2024-09-11T08:51:15.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 616913920 bytes 2024-09-11T08:51:15.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 654543872 bytes 2024-09-11T08:51:15.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 696685568 bytes 2024-09-11T08:51:15.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 750178304 bytes 2024-09-11T08:51:15.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 756922368 bytes 2024-09-11T08:51:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 776704000 bytes 2024-09-11T08:51:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 853543936 bytes 2024-09-11T08:51:16.248 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:51:16.248 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:51:16.249 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-11T08:51:16.249 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:51:16.249+0000 7f804c822640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 36761) UID: 1000 2024-09-11T08:51:16.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 910509056 bytes 2024-09-11T08:51:16.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 921515008 bytes 2024-09-11T08:51:16.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 964404224 bytes 2024-09-11T08:51:16.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1008797696 bytes 2024-09-11T08:51:16.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1041540096 bytes 2024-09-11T08:51:17.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1058932736 bytes 2024-09-11T08:51:17.249 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4385860036998084) > weight (0.25) 2024-09-11T08:51:17.249 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T08:51:17.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T08:51:17.250 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-11T08:51:17.250 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T08:51:17.249+0000 7ff9b6eb6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 37014) UID: 1000 2024-09-11T08:51:17.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1101649920 bytes 2024-09-11T08:51:17.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1121504256 bytes 2024-09-11T08:51:17.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1170573312 bytes 2024-09-11T08:51:17.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1202927616 bytes 2024-09-11T08:51:17.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1271666688 bytes 2024-09-11T08:51:18.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1281373184 bytes 2024-09-11T08:51:18.250 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:51:18.250 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:51:18.250 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:51:18.250 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:51:18.251 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:51:18.253 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:51:18.253 INFO:tasks.rbd_mirror_thrash:waiting for 85 secs before thrashing 2024-09-11T08:51:18.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1356761088 bytes 2024-09-11T08:51:18.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1399689216 bytes 2024-09-11T08:51:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1462487040 bytes 2024-09-11T08:51:18.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1463473152 bytes 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_2 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_2 snapshot 128M 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:18.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_2 128M 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:51:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:51:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:51:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:18.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_2 2024-09-11T08:51:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:51:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:51:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_2 snapshot 2024-09-11T08:51:18.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T08:51:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:18.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-11T08:51:19.250 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T08:51:19.251 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:51:19.251 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T08:51:19.251 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T08:51:19.251 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T08:51:19.254 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T08:51:19.254 INFO:tasks.rbd_mirror_thrash:waiting for 100 secs before thrashing 2024-09-11T08:51:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:51:19.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_2': 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1263f9d0fdc5 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1263f9d0fdc5 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:51:19.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:51:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:18 2024 2024-09-11T08:51:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:18 2024 2024-09-11T08:51:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:18 2024 2024-09-11T08:51:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:51:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:51:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 6176f630-2f7a-4485-9979-55a0a1132ec3 2024-09-11T08:51:19.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:51:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:51:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-11T08:51:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T08:51:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:51:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=13 2024-09-11T08:51:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:51:19.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:51:19.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 36377600 bytes 2024-09-11T08:51:19.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 36982784 bytes 2024-09-11T08:51:20.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 100999168 bytes 2024-09-11T08:51:20.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 119657472 bytes 2024-09-11T08:51:20.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 126550016 bytes 2024-09-11T08:51:20.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 167949312 bytes 2024-09-11T08:51:20.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 190535680 bytes 2024-09-11T08:51:20.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 272066560 bytes 2024-09-11T08:51:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 296039424 bytes 2024-09-11T08:51:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 341985280 bytes 2024-09-11T08:51:21.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 359406592 bytes 2024-09-11T08:51:21.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 409049088 bytes 2024-09-11T08:51:21.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 415465472 bytes 2024-09-11T08:51:21.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 471294976 bytes 2024-09-11T08:51:21.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 541673472 bytes 2024-09-11T08:51:22.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 560137216 bytes 2024-09-11T08:51:22.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 620050432 bytes 2024-09-11T08:51:22.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 650061824 bytes 2024-09-11T08:51:22.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 671328256 bytes 2024-09-11T08:51:22.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 729989120 bytes 2024-09-11T08:51:22.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 773046272 bytes 2024-09-11T08:51:23.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 801853440 bytes 2024-09-11T08:51:23.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 886503424 bytes 2024-09-11T08:51:23.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 932369408 bytes 2024-09-11T08:51:23.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1018781696 bytes 2024-09-11T08:51:23.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1027237888 bytes 2024-09-11T08:51:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1062880256 bytes 2024-09-11T08:51:24.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1072668672 bytes 2024-09-11T08:51:24.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1073283072 bytes 2024-09-11T08:51:24.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1083801600 bytes 2024-09-11T08:51:24.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1117146112 bytes 2024-09-11T08:51:24.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1165014016 bytes 2024-09-11T08:51:24.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1216835584 bytes 2024-09-11T08:51:25.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1267339264 bytes 2024-09-11T08:51:25.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1323492352 bytes 2024-09-11T08:51:25.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1362245632 bytes 2024-09-11T08:51:25.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1397196800 bytes 2024-09-11T08:51:25.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1410205696 bytes 2024-09-11T08:51:25.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1411927040 bytes 2024-09-11T08:51:25.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1497079808 bytes 2024-09-11T08:51:26.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1582797824 bytes 2024-09-11T08:51:26.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1633516544 bytes 2024-09-11T08:51:26.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1667213312 bytes 2024-09-11T08:51:26.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1684832256 bytes 2024-09-11T08:51:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1704667136 bytes 2024-09-11T08:51:26.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1714876416 bytes 2024-09-11T08:51:27.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1775881216 bytes 2024-09-11T08:51:27.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1795120128 bytes 2024-09-11T08:51:27.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1846144000 bytes 2024-09-11T08:51:27.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1888828416 bytes 2024-09-11T08:51:27.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 1907855360 bytes 2024-09-11T08:51:27.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 1942955008 bytes 2024-09-11T08:51:28.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2001137664 bytes 2024-09-11T08:51:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2030198784 bytes 2024-09-11T08:51:28.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2137688064 bytes 2024-09-11T08:51:28.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2171590656 bytes 2024-09-11T08:51:28.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2209478656 bytes 2024-09-11T08:51:28.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2235823104 bytes 2024-09-11T08:51:29.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2295240704 bytes 2024-09-11T08:51:29.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2331312128 bytes 2024-09-11T08:51:29.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2430096384 bytes 2024-09-11T08:51:29.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2494056448 bytes 2024-09-11T08:51:29.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2557179904 bytes 2024-09-11T08:51:30.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2559463424 bytes 2024-09-11T08:51:30.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2582960128 bytes 2024-09-11T08:51:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2618104832 bytes 2024-09-11T08:51:30.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2704880640 bytes 2024-09-11T08:51:30.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2738309120 bytes 2024-09-11T08:51:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2781438976 bytes 2024-09-11T08:51:31.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2806423552 bytes 2024-09-11T08:51:31.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2828880896 bytes 2024-09-11T08:51:31.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2834432000 bytes 2024-09-11T08:51:31.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 2869456896 bytes 2024-09-11T08:51:31.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 2889154560 bytes 2024-09-11T08:51:31.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 2914178048 bytes 2024-09-11T08:51:31.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 2986348544 bytes 2024-09-11T08:51:32.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7700 IOs, 3046994944 bytes 2024-09-11T08:51:32.376 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7800 IOs, 3100960768 bytes 2024-09-11T08:51:32.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7900 IOs, 3174753280 bytes 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_3 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_3 snapshot 128M 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:32.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_3 128M 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T08:51:32.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:51:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:51:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:51:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:32.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_3 2024-09-11T08:51:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:51:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:51:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:32.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_3 snapshot 2024-09-11T08:51:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T08:51:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:32.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-11T08:51:33.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:51:33.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-11T08:51:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_3': 2024-09-11T08:51:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:51:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:51:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:51:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12782397b4a3 2024-09-11T08:51:33.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12782397b4a3 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:32 2024 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:32 2024 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:32 2024 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8efd8144-a459-4554-b275-2ec378bad1af 2024-09-11T08:51:33.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:51:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:51:33.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-11T08:51:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T08:51:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:51:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=14 2024-09-11T08:51:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:51:33.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:51:33.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 17262592 bytes 2024-09-11T08:51:33.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 25995264 bytes 2024-09-11T08:51:34.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 88366080 bytes 2024-09-11T08:51:34.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 105901056 bytes 2024-09-11T08:51:34.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 171924480 bytes 2024-09-11T08:51:34.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 207475712 bytes 2024-09-11T08:51:34.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 248890368 bytes 2024-09-11T08:51:34.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 295364608 bytes 2024-09-11T08:51:35.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 339559424 bytes 2024-09-11T08:51:35.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 405647360 bytes 2024-09-11T08:51:35.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 411070464 bytes 2024-09-11T08:51:35.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 490246144 bytes 2024-09-11T08:51:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 514666496 bytes 2024-09-11T08:51:35.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 546805760 bytes 2024-09-11T08:51:36.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 576253952 bytes 2024-09-11T08:51:36.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 609448960 bytes 2024-09-11T08:51:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 641913856 bytes 2024-09-11T08:51:36.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 662692864 bytes 2024-09-11T08:51:36.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 703543296 bytes 2024-09-11T08:51:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 715310080 bytes 2024-09-11T08:51:36.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 755384320 bytes 2024-09-11T08:51:37.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 803088384 bytes 2024-09-11T08:51:37.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 815320064 bytes 2024-09-11T08:51:37.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 891542528 bytes 2024-09-11T08:51:37.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 923371520 bytes 2024-09-11T08:51:37.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 938525696 bytes 2024-09-11T08:51:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 992472064 bytes 2024-09-11T08:51:37.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 995007488 bytes 2024-09-11T08:51:38.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1005050880 bytes 2024-09-11T08:51:38.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1057157120 bytes 2024-09-11T08:51:38.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1069663232 bytes 2024-09-11T08:51:38.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1131315200 bytes 2024-09-11T08:51:38.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1182553088 bytes 2024-09-11T08:51:38.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1196023808 bytes 2024-09-11T08:51:38.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1238661120 bytes 2024-09-11T08:51:39.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1270611968 bytes 2024-09-11T08:51:39.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1277115392 bytes 2024-09-11T08:51:39.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1280617472 bytes 2024-09-11T08:51:39.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1309096960 bytes 2024-09-11T08:51:39.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1404047360 bytes 2024-09-11T08:51:39.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1431840768 bytes 2024-09-11T08:51:39.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1440290816 bytes 2024-09-11T08:51:40.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1505816576 bytes 2024-09-11T08:51:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1592144896 bytes 2024-09-11T08:51:40.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1608795136 bytes 2024-09-11T08:51:40.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1645641728 bytes 2024-09-11T08:51:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1739776000 bytes 2024-09-11T08:51:41.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1779600384 bytes 2024-09-11T08:51:41.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1820205056 bytes 2024-09-11T08:51:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1887727616 bytes 2024-09-11T08:51:41.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 1938617344 bytes 2024-09-11T08:51:41.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 1948784640 bytes 2024-09-11T08:51:41.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 1999403008 bytes 2024-09-11T08:51:41.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2000006144 bytes 2024-09-11T08:51:42.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2017469440 bytes 2024-09-11T08:51:42.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2039340032 bytes 2024-09-11T08:51:42.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2085724160 bytes 2024-09-11T08:51:42.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2088860672 bytes 2024-09-11T08:51:42.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2089473024 bytes 2024-09-11T08:51:42.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2158475264 bytes 2024-09-11T08:51:42.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2204353536 bytes 2024-09-11T08:51:43.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2229947392 bytes 2024-09-11T08:51:43.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2289027072 bytes 2024-09-11T08:51:43.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2304633856 bytes 2024-09-11T08:51:43.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2311098368 bytes 2024-09-11T08:51:43.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2353065984 bytes 2024-09-11T08:51:43.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2359389184 bytes 2024-09-11T08:51:43.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2359989248 bytes 2024-09-11T08:51:44.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2443516928 bytes 2024-09-11T08:51:44.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2486509568 bytes 2024-09-11T08:51:44.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2505720832 bytes 2024-09-11T08:51:44.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2539918336 bytes 2024-09-11T08:51:44.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 2552529920 bytes 2024-09-11T08:51:44.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 2671113216 bytes 2024-09-11T08:51:45.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 2684921856 bytes 2024-09-11T08:51:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 2687783936 bytes 2024-09-11T08:51:45.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7700 IOs, 2759856128 bytes 2024-09-11T08:51:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7800 IOs, 2836043776 bytes 2024-09-11T08:51:45.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7900 IOs, 2854574080 bytes 2024-09-11T08:51:45.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8000 IOs, 2889060352 bytes 2024-09-11T08:51:46.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8100 IOs, 2907230208 bytes 2024-09-11T08:51:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8200 IOs, 2948726784 bytes 2024-09-11T08:51:46.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8300 IOs, 2998178816 bytes 2024-09-11T08:51:46.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8400 IOs, 3032158208 bytes 2024-09-11T08:51:46.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8500 IOs, 3059869696 bytes 2024-09-11T08:51:46.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8600 IOs, 3060471808 bytes 2024-09-11T08:51:47.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8700 IOs, 3132374016 bytes 2024-09-11T08:51:47.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8800 IOs, 3203090432 bytes 2024-09-11T08:51:47.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:8900 IOs, 3209405440 bytes 2024-09-11T08:51:47.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:9000 IOs, 3249753088 bytes 2024-09-11T08:51:47.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:9100 IOs, 3284209664 bytes 2024-09-11T08:51:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:51:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:51:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:51:47.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_4 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_4 snapshot 128M 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:51:47.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_4 128M 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:51:47.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:47.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_4 2024-09-11T08:51:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:51:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:51:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_4 snapshot 2024-09-11T08:51:47.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T08:51:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:47.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-11T08:51:48.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:51:48.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-11T08:51:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_4': 2024-09-11T08:51:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:51:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:51:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:51:48.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1284c556f504 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1284c556f504 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:47 2024 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:47 2024 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:47 2024 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:51:48.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:51:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 39633655-9b79-45e6-ac71-541b46d2ea10 2024-09-11T08:51:48.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:51:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:51:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-11T08:51:48.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T08:51:48.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:51:48.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T08:51:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:51:48.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:51:48.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 28573696 bytes 2024-09-11T08:51:49.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 43888640 bytes 2024-09-11T08:51:49.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 62547968 bytes 2024-09-11T08:51:49.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 81020928 bytes 2024-09-11T08:51:49.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 139090944 bytes 2024-09-11T08:51:49.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 202757120 bytes 2024-09-11T08:51:50.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 236053504 bytes 2024-09-11T08:51:50.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 252951552 bytes 2024-09-11T08:51:50.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 269780992 bytes 2024-09-11T08:51:50.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 283244544 bytes 2024-09-11T08:51:50.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 283852800 bytes 2024-09-11T08:51:50.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 336388096 bytes 2024-09-11T08:51:50.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 372560896 bytes 2024-09-11T08:51:51.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 387548160 bytes 2024-09-11T08:51:51.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 444806144 bytes 2024-09-11T08:51:51.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 488663040 bytes 2024-09-11T08:51:51.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 529997824 bytes 2024-09-11T08:51:51.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 590100480 bytes 2024-09-11T08:51:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 607300608 bytes 2024-09-11T08:51:52.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 701529088 bytes 2024-09-11T08:51:52.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 743666688 bytes 2024-09-11T08:51:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 750942208 bytes 2024-09-11T08:51:52.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 780994560 bytes 2024-09-11T08:51:52.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 817006592 bytes 2024-09-11T08:51:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 872183808 bytes 2024-09-11T08:51:53.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 977319936 bytes 2024-09-11T08:51:53.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 989613056 bytes 2024-09-11T08:51:53.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1014572032 bytes 2024-09-11T08:51:53.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1046947840 bytes 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_5 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_5 snapshot 128M 2024-09-11T08:51:53.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_5 128M 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T08:51:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:51:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:51:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:51:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:51:53.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_5 2024-09-11T08:51:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:51:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:51:53.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:51:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_5 snapshot 2024-09-11T08:51:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T08:51:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:51:53.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-11T08:51:54.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:51:54.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_5': 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1290ab0fbe14 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1290ab0fbe14 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:51:54.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:53 2024 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:53 2024 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:53 2024 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 57b1ae5d-d195-4c6d-b9e5-3241f71e3de5 2024-09-11T08:51:54.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:51:54.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:51:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-11T08:51:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:51:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:51:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T08:51:54.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:51:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:51:54.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:51:54.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:51:55.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 76522496 bytes 2024-09-11T08:51:55.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 102983680 bytes 2024-09-11T08:51:55.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 162574336 bytes 2024-09-11T08:51:55.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 200969216 bytes 2024-09-11T08:51:55.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 211298304 bytes 2024-09-11T08:51:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 288011264 bytes 2024-09-11T08:51:56.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 312382464 bytes 2024-09-11T08:51:56.264 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 353825792 bytes 2024-09-11T08:51:56.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 393363456 bytes 2024-09-11T08:51:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 393945088 bytes 2024-09-11T08:51:56.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 405103616 bytes 2024-09-11T08:51:56.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 515165184 bytes 2024-09-11T08:51:57.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 574644224 bytes 2024-09-11T08:51:57.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 577301504 bytes 2024-09-11T08:51:57.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 648477696 bytes 2024-09-11T08:51:57.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 713609216 bytes 2024-09-11T08:51:57.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 760204288 bytes 2024-09-11T08:51:58.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 831128576 bytes 2024-09-11T08:51:58.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 858267648 bytes 2024-09-11T08:51:58.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 915827712 bytes 2024-09-11T08:51:58.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 961982464 bytes 2024-09-11T08:51:58.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 969944064 bytes 2024-09-11T08:51:58.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1033506816 bytes 2024-09-11T08:51:59.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1070129152 bytes 2024-09-11T08:51:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1084211200 bytes 2024-09-11T08:51:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1157114880 bytes 2024-09-11T08:51:59.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1201228800 bytes 2024-09-11T08:51:59.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1201835008 bytes 2024-09-11T08:51:59.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1215983616 bytes 2024-09-11T08:52:00.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1268386816 bytes 2024-09-11T08:52:00.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1302081536 bytes 2024-09-11T08:52:00.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1345756160 bytes 2024-09-11T08:52:00.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1359557632 bytes 2024-09-11T08:52:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1378911232 bytes 2024-09-11T08:52:00.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1441638400 bytes 2024-09-11T08:52:00.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1452355584 bytes 2024-09-11T08:52:01.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1528998912 bytes 2024-09-11T08:52:01.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1572852736 bytes 2024-09-11T08:52:01.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1612873728 bytes 2024-09-11T08:52:01.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1656582144 bytes 2024-09-11T08:52:01.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_6 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_6 snapshot 128M 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_6 128M 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:52:01.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:01.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_6 2024-09-11T08:52:01.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_6 snapshot 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-11T08:52:02.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:52:02.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-11T08:52:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_6': 2024-09-11T08:52:02.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 129c31be75fc 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.129c31be75fc 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:01 2024 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:01 2024 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:01 2024 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: bbfebdd5-c61c-4f17-9e2f-1d0a11cb2d73 2024-09-11T08:52:02.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:52:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:52:02.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-11T08:52:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T08:52:02.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:52:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=10 2024-09-11T08:52:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:52:02.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:52:02.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 17722368 bytes 2024-09-11T08:52:03.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 47473664 bytes 2024-09-11T08:52:03.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 118133760 bytes 2024-09-11T08:52:03.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 177037312 bytes 2024-09-11T08:52:03.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 248006656 bytes 2024-09-11T08:52:04.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 278615040 bytes 2024-09-11T08:52:04.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 302064640 bytes 2024-09-11T08:52:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 326195200 bytes 2024-09-11T08:52:04.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 362786816 bytes 2024-09-11T08:52:04.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 387878912 bytes 2024-09-11T08:52:04.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 443942912 bytes 2024-09-11T08:52:04.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 478091264 bytes 2024-09-11T08:52:05.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 507928576 bytes 2024-09-11T08:52:05.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 540179456 bytes 2024-09-11T08:52:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 582093824 bytes 2024-09-11T08:52:05.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 601624576 bytes 2024-09-11T08:52:05.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 626205696 bytes 2024-09-11T08:52:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 744525824 bytes 2024-09-11T08:52:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 758137856 bytes 2024-09-11T08:52:06.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 783370240 bytes 2024-09-11T08:52:06.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 813765632 bytes 2024-09-11T08:52:06.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 823655424 bytes 2024-09-11T08:52:06.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 852497408 bytes 2024-09-11T08:52:06.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 930362368 bytes 2024-09-11T08:52:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 956067840 bytes 2024-09-11T08:52:07.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 973604864 bytes 2024-09-11T08:52:07.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1015628800 bytes 2024-09-11T08:52:07.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1064822784 bytes 2024-09-11T08:52:07.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1113517056 bytes 2024-09-11T08:52:07.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1132536832 bytes 2024-09-11T08:52:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1133144064 bytes 2024-09-11T08:52:07.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1149582336 bytes 2024-09-11T08:52:07.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1150209024 bytes 2024-09-11T08:52:08.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1180827648 bytes 2024-09-11T08:52:08.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1212942336 bytes 2024-09-11T08:52:08.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1231506432 bytes 2024-09-11T08:52:08.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1268292608 bytes 2024-09-11T08:52:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1342139392 bytes 2024-09-11T08:52:09.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1419385856 bytes 2024-09-11T08:52:09.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1454023680 bytes 2024-09-11T08:52:09.365 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1507209216 bytes 2024-09-11T08:52:09.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1526616064 bytes 2024-09-11T08:52:09.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1577588736 bytes 2024-09-11T08:52:09.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1641986048 bytes 2024-09-11T08:52:10.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1660813312 bytes 2024-09-11T08:52:10.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1704933376 bytes 2024-09-11T08:52:10.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1715302400 bytes 2024-09-11T08:52:10.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1764978688 bytes 2024-09-11T08:52:10.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1815303168 bytes 2024-09-11T08:52:10.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1852319744 bytes 2024-09-11T08:52:11.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 1886119936 bytes 2024-09-11T08:52:11.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 1934402560 bytes 2024-09-11T08:52:11.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 1958036480 bytes 2024-09-11T08:52:11.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 1999205376 bytes 2024-09-11T08:52:11.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2021761024 bytes 2024-09-11T08:52:11.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2074038272 bytes 2024-09-11T08:52:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2173573120 bytes 2024-09-11T08:52:12.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2228108288 bytes 2024-09-11T08:52:12.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2296126464 bytes 2024-09-11T08:52:12.784 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2384625664 bytes 2024-09-11T08:52:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:52:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:52:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:52:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:52:12.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_7 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_7 snapshot 128M 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_7 128M 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:12.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_7 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_7 snapshot 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T08:52:12.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:12.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-11T08:52:13.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:52:13.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-11T08:52:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_7': 2024-09-11T08:52:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:52:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:52:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:52:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12a8571140eb 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12a8571140eb 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:12 2024 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:12 2024 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:12 2024 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 75596a83-10f1-4277-ad91-67db500695a7 2024-09-11T08:52:13.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:52:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:52:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-11T08:52:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T08:52:13.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:52:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T08:52:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:52:13.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:52:13.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 25536512 bytes 2024-09-11T08:52:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 38029312 bytes 2024-09-11T08:52:13.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 72975360 bytes 2024-09-11T08:52:13.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 97203200 bytes 2024-09-11T08:52:14.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 134897664 bytes 2024-09-11T08:52:14.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 226054144 bytes 2024-09-11T08:52:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 286259200 bytes 2024-09-11T08:52:14.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 305691648 bytes 2024-09-11T08:52:15.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 388215808 bytes 2024-09-11T08:52:15.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 388809728 bytes 2024-09-11T08:52:15.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 391061504 bytes 2024-09-11T08:52:15.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 410722304 bytes 2024-09-11T08:52:15.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 456893440 bytes 2024-09-11T08:52:15.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 457496576 bytes 2024-09-11T08:52:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 516187136 bytes 2024-09-11T08:52:16.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 530386944 bytes 2024-09-11T08:52:16.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 554014720 bytes 2024-09-11T08:52:16.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 597983232 bytes 2024-09-11T08:52:17.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 721910784 bytes 2024-09-11T08:52:17.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 755204096 bytes 2024-09-11T08:52:17.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 822819840 bytes 2024-09-11T08:52:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 869132288 bytes 2024-09-11T08:52:17.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 958977024 bytes 2024-09-11T08:52:17.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 987926528 bytes 2024-09-11T08:52:18.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1020039168 bytes 2024-09-11T08:52:18.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1066048512 bytes 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_8 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_8 snapshot 128M 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_8 128M 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:18.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_8 2024-09-11T08:52:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:52:18.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_8 snapshot 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:18.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-11T08:52:19.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:52:19.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-11T08:52:19.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_8': 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12b4822d1c7c 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12b4822d1c7c 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:18 2024 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:18 2024 2024-09-11T08:52:19.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:18 2024 2024-09-11T08:52:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:52:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:52:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 880a8fca-463b-4b43-b6d1-4b6ea06390b7 2024-09-11T08:52:19.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:52:19.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:52:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-11T08:52:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T08:52:19.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:52:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=8 2024-09-11T08:52:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:52:19.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:52:19.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 35342336 bytes 2024-09-11T08:52:19.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 100553728 bytes 2024-09-11T08:52:20.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 189559808 bytes 2024-09-11T08:52:20.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 215123968 bytes 2024-09-11T08:52:20.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 276689920 bytes 2024-09-11T08:52:20.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 295354368 bytes 2024-09-11T08:52:20.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 301163520 bytes 2024-09-11T08:52:20.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 362671104 bytes 2024-09-11T08:52:21.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 391425024 bytes 2024-09-11T08:52:21.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 437939200 bytes 2024-09-11T08:52:21.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 459277312 bytes 2024-09-11T08:52:21.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 510941184 bytes 2024-09-11T08:52:21.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 611591168 bytes 2024-09-11T08:52:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 628128768 bytes 2024-09-11T08:52:22.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 628745216 bytes 2024-09-11T08:52:22.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 661572608 bytes 2024-09-11T08:52:22.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 696489984 bytes 2024-09-11T08:52:22.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 731645952 bytes 2024-09-11T08:52:22.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 763443200 bytes 2024-09-11T08:52:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 786721792 bytes 2024-09-11T08:52:22.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 810923008 bytes 2024-09-11T08:52:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 856198144 bytes 2024-09-11T08:52:23.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 978690048 bytes 2024-09-11T08:52:23.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1010953216 bytes 2024-09-11T08:52:23.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1031843840 bytes 2024-09-11T08:52:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1072566272 bytes 2024-09-11T08:52:23.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1096035328 bytes 2024-09-11T08:52:24.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1106953216 bytes 2024-09-11T08:52:24.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1146338304 bytes 2024-09-11T08:52:24.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1170609152 bytes 2024-09-11T08:52:24.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1289094144 bytes 2024-09-11T08:52:25.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1371516928 bytes 2024-09-11T08:52:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1415037952 bytes 2024-09-11T08:52:25.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1437042688 bytes 2024-09-11T08:52:25.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1477659648 bytes 2024-09-11T08:52:25.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1503381504 bytes 2024-09-11T08:52:25.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1521771520 bytes 2024-09-11T08:52:26.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1526558720 bytes 2024-09-11T08:52:26.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1577772032 bytes 2024-09-11T08:52:26.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1592125440 bytes 2024-09-11T08:52:26.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1603018752 bytes 2024-09-11T08:52:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1661813760 bytes 2024-09-11T08:52:26.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1713296384 bytes 2024-09-11T08:52:26.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1713898496 bytes 2024-09-11T08:52:27.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1771581440 bytes 2024-09-11T08:52:27.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1821339648 bytes 2024-09-11T08:52:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:52:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:52:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:52:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:52:27.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_9 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_9 snapshot 128M 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_9 128M 2024-09-11T08:52:27.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:27.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_9 2024-09-11T08:52:27.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_9 snapshot 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:27.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-11T08:52:28.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:52:28.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_9': 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12c07a6c84e5 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12c07a6c84e5 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:52:28.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:27 2024 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:27 2024 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:27 2024 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 923d86f9-42b1-48a1-b1b7-848b1ff22bf5 2024-09-11T08:52:28.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:52:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:52:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-11T08:52:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T08:52:28.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:52:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=13 2024-09-11T08:52:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:52:28.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:52:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 38603776 bytes 2024-09-11T08:52:28.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 54157312 bytes 2024-09-11T08:52:29.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 151113728 bytes 2024-09-11T08:52:29.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 177051648 bytes 2024-09-11T08:52:29.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 270464000 bytes 2024-09-11T08:52:29.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 293810176 bytes 2024-09-11T08:52:29.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 339151872 bytes 2024-09-11T08:52:29.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 353153024 bytes 2024-09-11T08:52:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 387310592 bytes 2024-09-11T08:52:30.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 391328768 bytes 2024-09-11T08:52:30.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 413918208 bytes 2024-09-11T08:52:30.461 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 442472448 bytes 2024-09-11T08:52:30.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 462734336 bytes 2024-09-11T08:52:30.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 504885248 bytes 2024-09-11T08:52:30.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 555587584 bytes 2024-09-11T08:52:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 585899008 bytes 2024-09-11T08:52:31.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 598063104 bytes 2024-09-11T08:52:31.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 612260864 bytes 2024-09-11T08:52:31.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 644032512 bytes 2024-09-11T08:52:31.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 688531456 bytes 2024-09-11T08:52:31.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 730896384 bytes 2024-09-11T08:52:32.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 838855680 bytes 2024-09-11T08:52:32.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 850476032 bytes 2024-09-11T08:52:32.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 862311424 bytes 2024-09-11T08:52:32.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 894155776 bytes 2024-09-11T08:52:32.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 973809664 bytes 2024-09-11T08:52:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 974428160 bytes 2024-09-11T08:52:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 975045632 bytes 2024-09-11T08:52:33.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1027630080 bytes 2024-09-11T08:52:33.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1041282048 bytes 2024-09-11T08:52:33.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1051828224 bytes 2024-09-11T08:52:33.327 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:52:33.328+0000 7f00d11de640 -1 reset not still connected to 0x55b5ef0a7a00 2024-09-11T08:52:33.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1113691136 bytes 2024-09-11T08:52:33.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1208771584 bytes 2024-09-11T08:52:34.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1296022528 bytes 2024-09-11T08:52:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1361744896 bytes 2024-09-11T08:52:34.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1395800064 bytes 2024-09-11T08:52:34.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1413085184 bytes 2024-09-11T08:52:34.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1501059072 bytes 2024-09-11T08:52:34.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1518083072 bytes 2024-09-11T08:52:35.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1540784128 bytes 2024-09-11T08:52:35.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1562159104 bytes 2024-09-11T08:52:35.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1604294656 bytes 2024-09-11T08:52:35.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1644026880 bytes 2024-09-11T08:52:35.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1708383232 bytes 2024-09-11T08:52:35.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1728700416 bytes 2024-09-11T08:52:36.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1752830976 bytes 2024-09-11T08:52:36.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1815326720 bytes 2024-09-11T08:52:36.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1873689600 bytes 2024-09-11T08:52:36.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1964492800 bytes 2024-09-11T08:52:36.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2038120448 bytes 2024-09-11T08:52:37.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2073194496 bytes 2024-09-11T08:52:37.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2133985280 bytes 2024-09-11T08:52:37.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2188120064 bytes 2024-09-11T08:52:37.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2210253824 bytes 2024-09-11T08:52:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2219335680 bytes 2024-09-11T08:52:37.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2253075456 bytes 2024-09-11T08:52:38.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2325020672 bytes 2024-09-11T08:52:38.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2382047232 bytes 2024-09-11T08:52:38.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2419641344 bytes 2024-09-11T08:52:38.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2445402112 bytes 2024-09-11T08:52:38.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2518438912 bytes 2024-09-11T08:52:39.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2587130880 bytes 2024-09-11T08:52:39.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2661752832 bytes 2024-09-11T08:52:39.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2693856256 bytes 2024-09-11T08:52:39.613 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2748830720 bytes 2024-09-11T08:52:39.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2768343040 bytes 2024-09-11T08:52:39.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2768943104 bytes 2024-09-11T08:52:40.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2815993856 bytes 2024-09-11T08:52:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2833644544 bytes 2024-09-11T08:52:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2892316672 bytes 2024-09-11T08:52:40.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2923047936 bytes 2024-09-11T08:52:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2978129920 bytes 2024-09-11T08:52:40.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 3000924160 bytes 2024-09-11T08:52:40.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 3027644416 bytes 2024-09-11T08:52:41.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 3108217856 bytes 2024-09-11T08:52:41.364 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 3136521216 bytes 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_10 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_10 snapshot 128M 2024-09-11T08:52:41.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_10 128M 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:41.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T08:52:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:52:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:52:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:52:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:41.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_10 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_10 snapshot 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:41.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-11T08:52:42.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:52:42.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-11T08:52:42.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_10': 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12cc4c5e0be7 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12cc4c5e0be7 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:41 2024 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:41 2024 2024-09-11T08:52:42.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:41 2024 2024-09-11T08:52:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:52:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:52:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: b544cd07-41a2-4db4-b006-4225d4700d0a 2024-09-11T08:52:42.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:52:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:52:42.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-11T08:52:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T08:52:42.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:52:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=8 2024-09-11T08:52:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:52:42.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:52:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 66126848 bytes 2024-09-11T08:52:42.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 92418048 bytes 2024-09-11T08:52:43.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 174559232 bytes 2024-09-11T08:52:43.254 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:52:43.255 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:52:43.255 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-11T08:52:43.255 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:52:43.256 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:52:43.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:52:43.260+0000 7fc5d0dfd640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 38773) UID: 1000 2024-09-11T08:52:43.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 231258112 bytes 2024-09-11T08:52:43.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 280267776 bytes 2024-09-11T08:52:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 376397824 bytes 2024-09-11T08:52:44.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 399537152 bytes 2024-09-11T08:52:44.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 432806912 bytes 2024-09-11T08:52:44.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 435037184 bytes 2024-09-11T08:52:44.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 457684992 bytes 2024-09-11T08:52:44.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 509041664 bytes 2024-09-11T08:52:44.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 550168576 bytes 2024-09-11T08:52:44.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 565549056 bytes 2024-09-11T08:52:45.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 619380736 bytes 2024-09-11T08:52:45.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 649835520 bytes 2024-09-11T08:52:45.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 692411392 bytes 2024-09-11T08:52:45.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 737533952 bytes 2024-09-11T08:52:45.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 747346944 bytes 2024-09-11T08:52:45.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 786098176 bytes 2024-09-11T08:52:46.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 875151360 bytes 2024-09-11T08:52:46.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 890371072 bytes 2024-09-11T08:52:46.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 915985408 bytes 2024-09-11T08:52:46.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 973176832 bytes 2024-09-11T08:52:46.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1018246144 bytes 2024-09-11T08:52:46.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1054117888 bytes 2024-09-11T08:52:47.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1077420032 bytes 2024-09-11T08:52:47.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1096837120 bytes 2024-09-11T08:52:47.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1143216128 bytes 2024-09-11T08:52:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1180016640 bytes 2024-09-11T08:52:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1216021504 bytes 2024-09-11T08:52:47.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1291131904 bytes 2024-09-11T08:52:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1326353408 bytes 2024-09-11T08:52:48.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1362668544 bytes 2024-09-11T08:52:48.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1402494976 bytes 2024-09-11T08:52:48.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1419780096 bytes 2024-09-11T08:52:48.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1456783360 bytes 2024-09-11T08:52:48.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1501736960 bytes 2024-09-11T08:52:49.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1525629952 bytes 2024-09-11T08:52:49.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1555182592 bytes 2024-09-11T08:52:49.358 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:52:49.358 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:52:49.359 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:52:49.361 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:52:49.361 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-11T08:52:49.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1612063744 bytes 2024-09-11T08:52:49.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1643099136 bytes 2024-09-11T08:52:49.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1671196672 bytes 2024-09-11T08:52:49.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1693722624 bytes 2024-09-11T08:52:49.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1721050112 bytes 2024-09-11T08:52:50.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1752711168 bytes 2024-09-11T08:52:50.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1812795392 bytes 2024-09-11T08:52:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:52:50.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_11 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_11 snapshot 128M 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_11 128M 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T08:52:50.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:52:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:52:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:52:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:52:50.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_11 2024-09-11T08:52:50.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_11 snapshot 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:52:50.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_11 snapshot 2024-09-11T08:52:51.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:52:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_11 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_11': 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12dca3e9a904 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12dca3e9a904 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:52:51.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:52:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:50 2024 2024-09-11T08:52:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:50 2024 2024-09-11T08:52:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:50 2024 2024-09-11T08:52:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:52:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:52:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: e101a667-f9d8-4d05-a046-469b05b5b3df 2024-09-11T08:52:51.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:52:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:52:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_11 2024-09-11T08:52:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:52:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:52:51.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T08:52:51.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:52:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=12 2024-09-11T08:52:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:52:51.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_11 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:52:51.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 38130688 bytes 2024-09-11T08:52:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 38732800 bytes 2024-09-11T08:52:52.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 107011072 bytes 2024-09-11T08:52:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 143882240 bytes 2024-09-11T08:52:52.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 194255872 bytes 2024-09-11T08:52:52.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 216820736 bytes 2024-09-11T08:52:52.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 255076352 bytes 2024-09-11T08:52:52.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 288207872 bytes 2024-09-11T08:52:53.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 347598848 bytes 2024-09-11T08:52:53.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 462473216 bytes 2024-09-11T08:52:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 516634624 bytes 2024-09-11T08:52:53.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 627800064 bytes 2024-09-11T08:52:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 663049216 bytes 2024-09-11T08:52:54.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 723143680 bytes 2024-09-11T08:52:54.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 743686144 bytes 2024-09-11T08:52:54.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 776361984 bytes 2024-09-11T08:52:54.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 794065920 bytes 2024-09-11T08:52:54.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 840483840 bytes 2024-09-11T08:52:55.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 841100288 bytes 2024-09-11T08:52:55.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 853966848 bytes 2024-09-11T08:52:55.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 885036032 bytes 2024-09-11T08:52:55.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 957385728 bytes 2024-09-11T08:52:55.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 982039552 bytes 2024-09-11T08:52:55.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1018813440 bytes 2024-09-11T08:52:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1052859392 bytes 2024-09-11T08:52:56.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1141648384 bytes 2024-09-11T08:52:56.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1261371392 bytes 2024-09-11T08:52:56.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1261974528 bytes 2024-09-11T08:52:56.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1308640256 bytes 2024-09-11T08:52:57.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1379989504 bytes 2024-09-11T08:52:57.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1404487680 bytes 2024-09-11T08:52:57.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1446933504 bytes 2024-09-11T08:52:57.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1529069568 bytes 2024-09-11T08:52:57.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1542081536 bytes 2024-09-11T08:52:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1628439552 bytes 2024-09-11T08:52:58.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1642464256 bytes 2024-09-11T08:52:58.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1665433600 bytes 2024-09-11T08:52:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1762656256 bytes 2024-09-11T08:52:58.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1847902208 bytes 2024-09-11T08:52:58.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1873451008 bytes 2024-09-11T08:52:59.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1903717376 bytes 2024-09-11T08:52:59.254 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.457417381009937) > weight (0.25) 2024-09-11T08:52:59.255 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.5613519027295409) > weight (0.25) 2024-09-11T08:52:59.255 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.7093321190870784) > weight (0.25) 2024-09-11T08:52:59.255 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.47567616527177925) > weight (0.25) 2024-09-11T08:52:59.255 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-09-11T08:52:59.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1929573376 bytes 2024-09-11T08:52:59.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1938924544 bytes 2024-09-11T08:52:59.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1986031616 bytes 2024-09-11T08:52:59.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2037428224 bytes 2024-09-11T08:53:00.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2087383040 bytes 2024-09-11T08:53:00.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2200311808 bytes 2024-09-11T08:53:00.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2247953408 bytes 2024-09-11T08:53:00.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2311121920 bytes 2024-09-11T08:53:01.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2311737344 bytes 2024-09-11T08:53:01.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2337558528 bytes 2024-09-11T08:53:01.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2372614144 bytes 2024-09-11T08:53:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2432531456 bytes 2024-09-11T08:53:01.704 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2464465920 bytes 2024-09-11T08:53:01.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2540917760 bytes 2024-09-11T08:53:02.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2598771712 bytes 2024-09-11T08:53:02.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2686070784 bytes 2024-09-11T08:53:02.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2728927232 bytes 2024-09-11T08:53:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2774000640 bytes 2024-09-11T08:53:02.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2822226944 bytes 2024-09-11T08:53:03.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2888261632 bytes 2024-09-11T08:53:03.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2979871744 bytes 2024-09-11T08:53:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:53:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:53:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:53:03.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_12 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_12 snapshot 128M 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_12 128M 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:03.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_12 2024-09-11T08:53:03.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:53:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:53:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_12 snapshot 2024-09-11T08:53:03.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T08:53:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:03.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_12 snapshot 2024-09-11T08:53:04.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:53:04.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_12 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_12': 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12e8a8d8a096 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12e8a8d8a096 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:53:04.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:03 2024 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:03 2024 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:03 2024 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 0791c4fe-f9ee-44f1-8b14-208766525577 2024-09-11T08:53:04.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:53:04.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:53:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_12 2024-09-11T08:53:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T08:53:04.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:53:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=11 2024-09-11T08:53:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:53:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_12 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:53:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 64905216 bytes 2024-09-11T08:53:05.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 120649728 bytes 2024-09-11T08:53:05.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 165780480 bytes 2024-09-11T08:53:05.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 210461696 bytes 2024-09-11T08:53:05.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 275380224 bytes 2024-09-11T08:53:06.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 315009024 bytes 2024-09-11T08:53:06.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 373080064 bytes 2024-09-11T08:53:06.524 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 432384000 bytes 2024-09-11T08:53:06.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 504422400 bytes 2024-09-11T08:53:07.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 617730048 bytes 2024-09-11T08:53:07.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 627671040 bytes 2024-09-11T08:53:07.359 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 661578752 bytes 2024-09-11T08:53:07.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 737889280 bytes 2024-09-11T08:53:07.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 775810048 bytes 2024-09-11T08:53:08.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 838838272 bytes 2024-09-11T08:53:08.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 851536896 bytes 2024-09-11T08:53:08.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 918635520 bytes 2024-09-11T08:53:08.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 966877184 bytes 2024-09-11T08:53:08.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1064987648 bytes 2024-09-11T08:53:09.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1093014528 bytes 2024-09-11T08:53:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1117481984 bytes 2024-09-11T08:53:09.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1138668544 bytes 2024-09-11T08:53:09.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1222426624 bytes 2024-09-11T08:53:09.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1260966912 bytes 2024-09-11T08:53:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1276774400 bytes 2024-09-11T08:53:10.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1286145024 bytes 2024-09-11T08:53:10.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1330039808 bytes 2024-09-11T08:53:10.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1415639040 bytes 2024-09-11T08:53:10.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1470844928 bytes 2024-09-11T08:53:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1546203136 bytes 2024-09-11T08:53:11.259 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1585294336 bytes 2024-09-11T08:53:11.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1641612288 bytes 2024-09-11T08:53:11.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1712477184 bytes 2024-09-11T08:53:11.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1761862656 bytes 2024-09-11T08:53:12.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1809579008 bytes 2024-09-11T08:53:12.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1861664768 bytes 2024-09-11T08:53:12.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1906704384 bytes 2024-09-11T08:53:12.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1951233024 bytes 2024-09-11T08:53:12.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1998260224 bytes 2024-09-11T08:53:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 2037245952 bytes 2024-09-11T08:53:13.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 2089512960 bytes 2024-09-11T08:53:13.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 2104275968 bytes 2024-09-11T08:53:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 2116819968 bytes 2024-09-11T08:53:13.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 2147210240 bytes 2024-09-11T08:53:13.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2162680832 bytes 2024-09-11T08:53:13.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2212783104 bytes 2024-09-11T08:53:14.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2259644416 bytes 2024-09-11T08:53:14.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2336116736 bytes 2024-09-11T08:53:14.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2339381248 bytes 2024-09-11T08:53:14.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2394234880 bytes 2024-09-11T08:53:14.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2429773824 bytes 2024-09-11T08:53:14.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2506299392 bytes 2024-09-11T08:53:15.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2555499520 bytes 2024-09-11T08:53:15.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2600630272 bytes 2024-09-11T08:53:15.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2623740928 bytes 2024-09-11T08:53:15.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2631341056 bytes 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_13 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_13 snapshot 128M 2024-09-11T08:53:15.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_13 128M 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:15.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T08:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:15.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_13 2024-09-11T08:53:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:53:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:53:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:15.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_13 snapshot 2024-09-11T08:53:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T08:53:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:15.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_13 snapshot 2024-09-11T08:53:16.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:53:16.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_13 2024-09-11T08:53:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_13': 2024-09-11T08:53:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:53:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:53:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:53:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12f4df786b41 2024-09-11T08:53:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12f4df786b41 2024-09-11T08:53:16.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:15 2024 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:15 2024 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:15 2024 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 551a4abb-edcd-4eb2-8d3f-d80000f55c42 2024-09-11T08:53:16.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:53:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:53:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_13 2024-09-11T08:53:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:16.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T08:53:16.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:53:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T08:53:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:53:16.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_13 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:53:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 632832 bytes 2024-09-11T08:53:17.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 82008064 bytes 2024-09-11T08:53:17.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 88287232 bytes 2024-09-11T08:53:17.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 134866944 bytes 2024-09-11T08:53:17.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 155881472 bytes 2024-09-11T08:53:17.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 158712832 bytes 2024-09-11T08:53:17.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 186408960 bytes 2024-09-11T08:53:18.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 279749632 bytes 2024-09-11T08:53:18.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 333124608 bytes 2024-09-11T08:53:18.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 354219008 bytes 2024-09-11T08:53:18.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 378055680 bytes 2024-09-11T08:53:18.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 404483072 bytes 2024-09-11T08:53:18.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 463244288 bytes 2024-09-11T08:53:19.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 514574336 bytes 2024-09-11T08:53:19.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 665850880 bytes 2024-09-11T08:53:19.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 679030784 bytes 2024-09-11T08:53:19.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 722323456 bytes 2024-09-11T08:53:20.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 758322176 bytes 2024-09-11T08:53:20.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 804706304 bytes 2024-09-11T08:53:20.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 838051840 bytes 2024-09-11T08:53:20.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 877698048 bytes 2024-09-11T08:53:20.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 884111360 bytes 2024-09-11T08:53:20.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 912002048 bytes 2024-09-11T08:53:21.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 941010944 bytes 2024-09-11T08:53:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 957651968 bytes 2024-09-11T08:53:21.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 985310208 bytes 2024-09-11T08:53:21.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 985920512 bytes 2024-09-11T08:53:21.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1052459008 bytes 2024-09-11T08:53:21.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_14 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_14 snapshot 128M 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_14 128M 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_14 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_14 snapshot 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T08:53:21.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:21.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_14 snapshot 2024-09-11T08:53:22.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:53:22.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_14 2024-09-11T08:53:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_14': 2024-09-11T08:53:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:53:22.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1300fd85d1ab 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1300fd85d1ab 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:21 2024 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:21 2024 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:21 2024 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:53:22.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 4174ebf4-c3c8-4e27-a392-9c68a9fb6a29 2024-09-11T08:53:22.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:53:22.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:53:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_14 2024-09-11T08:53:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T08:53:22.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:53:22.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T08:53:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:53:22.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_14 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:53:23.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 120072192 bytes 2024-09-11T08:53:23.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 132274176 bytes 2024-09-11T08:53:23.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 145424384 bytes 2024-09-11T08:53:23.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 151284736 bytes 2024-09-11T08:53:23.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 211422208 bytes 2024-09-11T08:53:23.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 231185408 bytes 2024-09-11T08:53:24.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 288522240 bytes 2024-09-11T08:53:24.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 319066112 bytes 2024-09-11T08:53:24.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 341181440 bytes 2024-09-11T08:53:24.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 353616896 bytes 2024-09-11T08:53:24.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 403749888 bytes 2024-09-11T08:53:24.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 482342912 bytes 2024-09-11T08:53:25.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 557128704 bytes 2024-09-11T08:53:25.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 635037696 bytes 2024-09-11T08:53:25.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 689991680 bytes 2024-09-11T08:53:25.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 752809984 bytes 2024-09-11T08:53:25.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 779855872 bytes 2024-09-11T08:53:26.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 820913152 bytes 2024-09-11T08:53:26.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 842527744 bytes 2024-09-11T08:53:26.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 850849792 bytes 2024-09-11T08:53:26.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 906465280 bytes 2024-09-11T08:53:26.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 907071488 bytes 2024-09-11T08:53:26.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 953034752 bytes 2024-09-11T08:53:27.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1039372288 bytes 2024-09-11T08:53:27.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1102432256 bytes 2024-09-11T08:53:27.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1218134016 bytes 2024-09-11T08:53:27.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1249599488 bytes 2024-09-11T08:53:28.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1318769664 bytes 2024-09-11T08:53:28.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1346662400 bytes 2024-09-11T08:53:28.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1426832384 bytes 2024-09-11T08:53:28.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1443872768 bytes 2024-09-11T08:53:28.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1459193856 bytes 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_15 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_15 snapshot 128M 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:28.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_15 128M 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:53:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:53:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:53:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:28.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_15 2024-09-11T08:53:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:53:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:53:28.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_15 snapshot 2024-09-11T08:53:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T08:53:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:28.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_15 snapshot 2024-09-11T08:53:29.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:53:29.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_15 2024-09-11T08:53:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_15': 2024-09-11T08:53:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:53:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:53:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:53:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 130c78585bf7 2024-09-11T08:53:29.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.130c78585bf7 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:28 2024 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:28 2024 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:28 2024 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 5f90212f-9abd-40ef-98b3-fab670ebcab5 2024-09-11T08:53:29.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:53:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:53:29.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_15 2024-09-11T08:53:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T08:53:29.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:53:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=14 2024-09-11T08:53:29.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:53:29.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_15 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:53:30.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 30729216 bytes 2024-09-11T08:53:30.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 116511744 bytes 2024-09-11T08:53:30.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 175586304 bytes 2024-09-11T08:53:30.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 307337216 bytes 2024-09-11T08:53:31.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 389287936 bytes 2024-09-11T08:53:31.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 432750592 bytes 2024-09-11T08:53:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 441436160 bytes 2024-09-11T08:53:31.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 490165248 bytes 2024-09-11T08:53:31.671 INFO:tasks.ceph.cluster2.osd.1.smithi077.stderr:2024-09-11T08:53:31.674+0000 7f35e8ea7640 -1 reset not still connected to 0x5595f9ea5ba0 2024-09-11T08:53:31.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 540690432 bytes 2024-09-11T08:53:31.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 589822976 bytes 2024-09-11T08:53:32.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 652218368 bytes 2024-09-11T08:53:32.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 724901888 bytes 2024-09-11T08:53:32.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 749647872 bytes 2024-09-11T08:53:32.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 773331968 bytes 2024-09-11T08:53:32.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 811683840 bytes 2024-09-11T08:53:33.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 867736576 bytes 2024-09-11T08:53:33.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 909268992 bytes 2024-09-11T08:53:33.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 909849600 bytes 2024-09-11T08:53:33.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 932333568 bytes 2024-09-11T08:53:33.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1011156992 bytes 2024-09-11T08:53:33.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1063909376 bytes 2024-09-11T08:53:34.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1139781632 bytes 2024-09-11T08:53:34.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1176327168 bytes 2024-09-11T08:53:34.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1230923776 bytes 2024-09-11T08:53:34.785 INFO:tasks.ceph.cluster2.osd.0.smithi077.stderr:2024-09-11T08:53:34.786+0000 7f2b03c68640 -1 reset not still connected to 0x55c6037c9380 2024-09-11T08:53:34.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1261089792 bytes 2024-09-11T08:53:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1296753664 bytes 2024-09-11T08:53:35.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1336855552 bytes 2024-09-11T08:53:35.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1362367488 bytes 2024-09-11T08:53:35.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1362997248 bytes 2024-09-11T08:53:35.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1369448448 bytes 2024-09-11T08:53:35.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1401870336 bytes 2024-09-11T08:53:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1402483712 bytes 2024-09-11T08:53:36.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1430002688 bytes 2024-09-11T08:53:36.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1456048128 bytes 2024-09-11T08:53:36.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1507381248 bytes 2024-09-11T08:53:36.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1526033408 bytes 2024-09-11T08:53:36.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1559424000 bytes 2024-09-11T08:53:36.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1599356928 bytes 2024-09-11T08:53:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1660735488 bytes 2024-09-11T08:53:37.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1669481472 bytes 2024-09-11T08:53:37.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1724387328 bytes 2024-09-11T08:53:37.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1754712064 bytes 2024-09-11T08:53:37.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1765230592 bytes 2024-09-11T08:53:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1792312320 bytes 2024-09-11T08:53:37.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1824281600 bytes 2024-09-11T08:53:37.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1831140352 bytes 2024-09-11T08:53:38.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1861208064 bytes 2024-09-11T08:53:38.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1894946816 bytes 2024-09-11T08:53:38.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1948004352 bytes 2024-09-11T08:53:38.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1986143232 bytes 2024-09-11T08:53:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2011236352 bytes 2024-09-11T08:53:39.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2061724672 bytes 2024-09-11T08:53:39.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2121981952 bytes 2024-09-11T08:53:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2148853760 bytes 2024-09-11T08:53:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2193915904 bytes 2024-09-11T08:53:39.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2240680960 bytes 2024-09-11T08:53:39.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2297083904 bytes 2024-09-11T08:53:40.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2312183808 bytes 2024-09-11T08:53:40.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2382150656 bytes 2024-09-11T08:53:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2382761984 bytes 2024-09-11T08:53:40.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2383398912 bytes 2024-09-11T08:53:40.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2423428096 bytes 2024-09-11T08:53:41.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2563192832 bytes 2024-09-11T08:53:41.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2594849792 bytes 2024-09-11T08:53:41.376 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2652066816 bytes 2024-09-11T08:53:41.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2668276736 bytes 2024-09-11T08:53:41.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2699787264 bytes 2024-09-11T08:53:41.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2749280256 bytes 2024-09-11T08:53:42.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2835123200 bytes 2024-09-11T08:53:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2850446336 bytes 2024-09-11T08:53:42.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2925577216 bytes 2024-09-11T08:53:42.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2980375552 bytes 2024-09-11T08:53:42.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 3007849472 bytes 2024-09-11T08:53:43.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 3100474368 bytes 2024-09-11T08:53:43.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 3128101888 bytes 2024-09-11T08:53:43.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 3160752128 bytes 2024-09-11T08:53:43.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7700 IOs, 3189733376 bytes 2024-09-11T08:53:43.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7800 IOs, 3190337536 bytes 2024-09-11T08:53:43.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7900 IOs, 3259948032 bytes 2024-09-11T08:53:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:53:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:53:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:53:43.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_16 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_16 snapshot 128M 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_16 128M 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:43.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_16 2024-09-11T08:53:43.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_16 snapshot 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:43.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_16 snapshot 2024-09-11T08:53:44.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:53:44.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_16 2024-09-11T08:53:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_16': 2024-09-11T08:53:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:53:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:53:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:53:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 131830c8be36 2024-09-11T08:53:44.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.131830c8be36 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:43 2024 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:43 2024 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:43 2024 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 82fa71ee-6b32-459f-a37e-db8386916102 2024-09-11T08:53:44.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:53:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:53:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_16 2024-09-11T08:53:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T08:53:44.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:53:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:53:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:53:44.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_16 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:53:44.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 2952192 bytes 2024-09-11T08:53:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 32647168 bytes 2024-09-11T08:53:45.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 155311104 bytes 2024-09-11T08:53:45.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 155918336 bytes 2024-09-11T08:53:45.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 197882880 bytes 2024-09-11T08:53:45.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 239010816 bytes 2024-09-11T08:53:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 262250496 bytes 2024-09-11T08:53:45.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 287457280 bytes 2024-09-11T08:53:46.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 357163008 bytes 2024-09-11T08:53:46.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 454674432 bytes 2024-09-11T08:53:46.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 530708480 bytes 2024-09-11T08:53:46.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 553848832 bytes 2024-09-11T08:53:47.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 622593024 bytes 2024-09-11T08:53:47.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 653712384 bytes 2024-09-11T08:53:47.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 654327808 bytes 2024-09-11T08:53:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 672706560 bytes 2024-09-11T08:53:47.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 741268480 bytes 2024-09-11T08:53:47.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 840169472 bytes 2024-09-11T08:53:48.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 895248384 bytes 2024-09-11T08:53:48.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 942032896 bytes 2024-09-11T08:53:48.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1018654720 bytes 2024-09-11T08:53:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1044175872 bytes 2024-09-11T08:53:48.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1061724160 bytes 2024-09-11T08:53:48.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1122063360 bytes 2024-09-11T08:53:49.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1172009984 bytes 2024-09-11T08:53:49.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1245684736 bytes 2024-09-11T08:53:49.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1281120256 bytes 2024-09-11T08:53:49.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1292706816 bytes 2024-09-11T08:53:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1372147712 bytes 2024-09-11T08:53:50.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1411179520 bytes 2024-09-11T08:53:50.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1411798016 bytes 2024-09-11T08:53:50.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1457821696 bytes 2024-09-11T08:53:50.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1517729792 bytes 2024-09-11T08:53:50.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1568192512 bytes 2024-09-11T08:53:50.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1570432000 bytes 2024-09-11T08:53:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1618772992 bytes 2024-09-11T08:53:51.259 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1654163456 bytes 2024-09-11T08:53:51.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1698497536 bytes 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_17 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_17 snapshot 128M 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_17 128M 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:53:51.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:51.519 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_17 2024-09-11T08:53:51.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_17 snapshot 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:51.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_17 snapshot 2024-09-11T08:53:52.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:53:52.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_17 2024-09-11T08:53:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_17': 2024-09-11T08:53:52.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1324c1ad23a0 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1324c1ad23a0 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:51 2024 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:51 2024 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:51 2024 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 28968644-b7f0-40d0-b9ce-2784d7675298 2024-09-11T08:53:52.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:53:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:53:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_17 2024-09-11T08:53:52.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T08:53:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:53:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T08:53:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:53:52.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_17 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:53:52.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 54169600 bytes 2024-09-11T08:53:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 86116352 bytes 2024-09-11T08:53:53.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 111327232 bytes 2024-09-11T08:53:53.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 149433344 bytes 2024-09-11T08:53:53.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 196295680 bytes 2024-09-11T08:53:53.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 253264896 bytes 2024-09-11T08:53:53.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 327240704 bytes 2024-09-11T08:53:53.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 339906560 bytes 2024-09-11T08:53:54.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 387245056 bytes 2024-09-11T08:53:54.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 387833856 bytes 2024-09-11T08:53:54.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 424412160 bytes 2024-09-11T08:53:54.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 444435456 bytes 2024-09-11T08:53:54.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 469189632 bytes 2024-09-11T08:53:54.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 491683840 bytes 2024-09-11T08:53:54.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 528604160 bytes 2024-09-11T08:53:55.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 556734464 bytes 2024-09-11T08:53:55.259 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 596339712 bytes 2024-09-11T08:53:55.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 641975296 bytes 2024-09-11T08:53:55.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 725580800 bytes 2024-09-11T08:53:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 756237312 bytes 2024-09-11T08:53:55.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 761808896 bytes 2024-09-11T08:53:56.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 783721472 bytes 2024-09-11T08:53:56.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 855589888 bytes 2024-09-11T08:53:56.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 867490816 bytes 2024-09-11T08:53:56.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 894983168 bytes 2024-09-11T08:53:56.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 985608192 bytes 2024-09-11T08:53:57.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1033684992 bytes 2024-09-11T08:53:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1077174272 bytes 2024-09-11T08:53:57.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1144039424 bytes 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_18 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_18 snapshot 128M 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:57.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_18 128M 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:53:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:53:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:53:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:53:57.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_18 2024-09-11T08:53:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:53:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:53:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:53:57.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_18 snapshot 2024-09-11T08:53:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T08:53:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:53:57.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_18 snapshot 2024-09-11T08:53:58.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:53:58.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_18 2024-09-11T08:53:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_18': 2024-09-11T08:53:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:53:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:53:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:53:58.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13305b77790e 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13305b77790e 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:57 2024 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:57 2024 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:57 2024 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 350bf260-3a2c-48eb-8813-d7dafe84bdb1 2024-09-11T08:53:58.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:53:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:53:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_18 2024-09-11T08:53:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:53:58.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:53:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T08:53:58.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:53:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T08:53:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:53:58.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_18 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:53:58.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 14845952 bytes 2024-09-11T08:53:58.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 42777600 bytes 2024-09-11T08:53:58.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 57865216 bytes 2024-09-11T08:53:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 98892800 bytes 2024-09-11T08:53:59.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 144419840 bytes 2024-09-11T08:53:59.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 190863360 bytes 2024-09-11T08:53:59.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 230511616 bytes 2024-09-11T08:53:59.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 257107968 bytes 2024-09-11T08:53:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 284409856 bytes 2024-09-11T08:54:00.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 352201728 bytes 2024-09-11T08:54:00.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 391678976 bytes 2024-09-11T08:54:00.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 440624128 bytes 2024-09-11T08:54:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 518312960 bytes 2024-09-11T08:54:00.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 529081344 bytes 2024-09-11T08:54:01.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 541294592 bytes 2024-09-11T08:54:01.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 567655424 bytes 2024-09-11T08:54:01.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 639296512 bytes 2024-09-11T08:54:01.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 703279104 bytes 2024-09-11T08:54:01.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 783465472 bytes 2024-09-11T08:54:01.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 784096256 bytes 2024-09-11T08:54:02.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 807766016 bytes 2024-09-11T08:54:02.257 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5500229296491546) > weight (0.25) 2024-09-11T08:54:02.257 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.844606722116728) > weight (0.25) 2024-09-11T08:54:02.257 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.45287425385488356) > weight (0.25) 2024-09-11T08:54:02.257 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7956716522120059) > weight (0.25) 2024-09-11T08:54:02.257 INFO:tasks.rbd_mirror_thrash:waiting for 63 secs before thrashing 2024-09-11T08:54:02.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 851931136 bytes 2024-09-11T08:54:02.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 922597376 bytes 2024-09-11T08:54:02.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 987056128 bytes 2024-09-11T08:54:03.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1065537536 bytes 2024-09-11T08:54:03.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1117440000 bytes 2024-09-11T08:54:03.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1187734528 bytes 2024-09-11T08:54:03.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1322166272 bytes 2024-09-11T08:54:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1368247296 bytes 2024-09-11T08:54:04.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1456040960 bytes 2024-09-11T08:54:04.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1495088128 bytes 2024-09-11T08:54:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:54:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:54:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:54:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:54:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_19 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_19 snapshot 128M 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_19 128M 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:04.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_19 2024-09-11T08:54:04.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_19 snapshot 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:04.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_19 snapshot 2024-09-11T08:54:05.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:54:05.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_19 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_19': 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 133cd84cc6b6 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.133cd84cc6b6 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:54:05.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:04 2024 2024-09-11T08:54:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:04 2024 2024-09-11T08:54:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:04 2024 2024-09-11T08:54:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:54:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:54:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 05c8607a-0bf6-407e-b458-e04320b3a7ac 2024-09-11T08:54:05.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:54:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:54:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_19 2024-09-11T08:54:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T08:54:05.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:54:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=13 2024-09-11T08:54:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:54:05.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_19 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:54:05.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 62748672 bytes 2024-09-11T08:54:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 82236416 bytes 2024-09-11T08:54:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 162395136 bytes 2024-09-11T08:54:06.461 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 262029312 bytes 2024-09-11T08:54:06.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 331022336 bytes 2024-09-11T08:54:06.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 339962880 bytes 2024-09-11T08:54:06.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 379624448 bytes 2024-09-11T08:54:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 407359488 bytes 2024-09-11T08:54:07.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 511420416 bytes 2024-09-11T08:54:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 563924992 bytes 2024-09-11T08:54:07.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 634777600 bytes 2024-09-11T08:54:07.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 693127168 bytes 2024-09-11T08:54:08.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 756658176 bytes 2024-09-11T08:54:08.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 778728448 bytes 2024-09-11T08:54:08.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 827794432 bytes 2024-09-11T08:54:08.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 904105984 bytes 2024-09-11T08:54:08.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 942795776 bytes 2024-09-11T08:54:09.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 996601856 bytes 2024-09-11T08:54:09.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1039168512 bytes 2024-09-11T08:54:09.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1069509632 bytes 2024-09-11T08:54:09.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1078887424 bytes 2024-09-11T08:54:09.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1112383488 bytes 2024-09-11T08:54:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1133154304 bytes 2024-09-11T08:54:10.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1145698304 bytes 2024-09-11T08:54:10.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1146300416 bytes 2024-09-11T08:54:10.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1168198656 bytes 2024-09-11T08:54:10.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1228279808 bytes 2024-09-11T08:54:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1273774080 bytes 2024-09-11T08:54:10.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1308863488 bytes 2024-09-11T08:54:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1329960960 bytes 2024-09-11T08:54:11.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1340653568 bytes 2024-09-11T08:54:11.253 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1407148032 bytes 2024-09-11T08:54:11.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1423179776 bytes 2024-09-11T08:54:11.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1468703744 bytes 2024-09-11T08:54:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1513449472 bytes 2024-09-11T08:54:11.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1539188736 bytes 2024-09-11T08:54:11.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1539799040 bytes 2024-09-11T08:54:12.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1567783936 bytes 2024-09-11T08:54:12.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1586714624 bytes 2024-09-11T08:54:12.400 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1612574720 bytes 2024-09-11T08:54:12.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1648889856 bytes 2024-09-11T08:54:12.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1745059840 bytes 2024-09-11T08:54:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1817701376 bytes 2024-09-11T08:54:13.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1865261056 bytes 2024-09-11T08:54:13.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1908537344 bytes 2024-09-11T08:54:13.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1951086592 bytes 2024-09-11T08:54:13.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2034419712 bytes 2024-09-11T08:54:13.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2040634368 bytes 2024-09-11T08:54:14.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2072869888 bytes 2024-09-11T08:54:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2115633152 bytes 2024-09-11T08:54:14.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2133089280 bytes 2024-09-11T08:54:14.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2168188928 bytes 2024-09-11T08:54:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2180946944 bytes 2024-09-11T08:54:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2229839872 bytes 2024-09-11T08:54:15.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2282744832 bytes 2024-09-11T08:54:15.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2283372544 bytes 2024-09-11T08:54:15.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2299908096 bytes 2024-09-11T08:54:15.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.32784816597361466) > weight (0.25) 2024-09-11T08:54:15.363 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.7656816084624211) > weight (0.25) 2024-09-11T08:54:15.363 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T08:54:15.363 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T08:54:15.364 INFO:tasks.rbd_mirror_thrash:waiting for 5 secs before reviving daemons 2024-09-11T08:54:15.364 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T08:54:15.365+0000 7fde65a66640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 34810) UID: 1000 2024-09-11T08:54:15.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2345703424 bytes 2024-09-11T08:54:15.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2361349120 bytes 2024-09-11T08:54:15.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2400493568 bytes 2024-09-11T08:54:15.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2426896384 bytes 2024-09-11T08:54:16.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2464837632 bytes 2024-09-11T08:54:16.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2533089280 bytes 2024-09-11T08:54:16.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2533698560 bytes 2024-09-11T08:54:16.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2620084224 bytes 2024-09-11T08:54:16.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2694689792 bytes 2024-09-11T08:54:17.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2737346560 bytes 2024-09-11T08:54:17.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2776127488 bytes 2024-09-11T08:54:17.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2812262400 bytes 2024-09-11T08:54:17.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2881587200 bytes 2024-09-11T08:54:17.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2951923712 bytes 2024-09-11T08:54:18.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2960054272 bytes 2024-09-11T08:54:18.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 3015388160 bytes 2024-09-11T08:54:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 3064957952 bytes 2024-09-11T08:54:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:54:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:54:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:54:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:18.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_20 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_20 snapshot 128M 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_20 128M 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:18.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_20 2024-09-11T08:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_20 snapshot 2024-09-11T08:54:18.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T08:54:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:18.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_20 snapshot 2024-09-11T08:54:19.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:54:19.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_20 2024-09-11T08:54:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_20': 2024-09-11T08:54:19.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 134745c7edd4 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.134745c7edd4 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:18 2024 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:18 2024 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:18 2024 2024-09-11T08:54:19.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:54:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:54:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 6bae77b4-74ad-4d05-87e9-78bd958ad83a 2024-09-11T08:54:19.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:54:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:54:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_20 2024-09-11T08:54:19.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T08:54:19.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:54:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:54:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:54:19.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_20 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:54:19.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 26846208 bytes 2024-09-11T08:54:20.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 76003328 bytes 2024-09-11T08:54:20.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 154875904 bytes 2024-09-11T08:54:20.364 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T08:54:20.364 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:54:20.365 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T08:54:20.365 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T08:54:20.365 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T08:54:20.369 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T08:54:20.369 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-11T08:54:20.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 214992896 bytes 2024-09-11T08:54:20.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 246348800 bytes 2024-09-11T08:54:20.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 294626304 bytes 2024-09-11T08:54:21.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 327580672 bytes 2024-09-11T08:54:21.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 347607040 bytes 2024-09-11T08:54:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 394364928 bytes 2024-09-11T08:54:21.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 434110464 bytes 2024-09-11T08:54:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 455092224 bytes 2024-09-11T08:54:21.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 486909952 bytes 2024-09-11T08:54:22.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 523999232 bytes 2024-09-11T08:54:22.268 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 553831424 bytes 2024-09-11T08:54:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 559336448 bytes 2024-09-11T08:54:22.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 628538368 bytes 2024-09-11T08:54:22.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 635483136 bytes 2024-09-11T08:54:22.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 656324608 bytes 2024-09-11T08:54:23.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 698283008 bytes 2024-09-11T08:54:23.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 723650560 bytes 2024-09-11T08:54:23.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 792017920 bytes 2024-09-11T08:54:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 851245056 bytes 2024-09-11T08:54:23.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 889150464 bytes 2024-09-11T08:54:23.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 947270656 bytes 2024-09-11T08:54:24.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 998061056 bytes 2024-09-11T08:54:24.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1058012160 bytes 2024-09-11T08:54:24.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1083957248 bytes 2024-09-11T08:54:24.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1091521536 bytes 2024-09-11T08:54:24.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1098853376 bytes 2024-09-11T08:54:25.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1196495872 bytes 2024-09-11T08:54:25.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1225728000 bytes 2024-09-11T08:54:25.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1250959360 bytes 2024-09-11T08:54:25.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1292552192 bytes 2024-09-11T08:54:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1364759552 bytes 2024-09-11T08:54:25.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1399297024 bytes 2024-09-11T08:54:26.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1465104384 bytes 2024-09-11T08:54:26.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1469670400 bytes 2024-09-11T08:54:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1480617984 bytes 2024-09-11T08:54:26.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1532014592 bytes 2024-09-11T08:54:26.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1582342144 bytes 2024-09-11T08:54:26.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1635623936 bytes 2024-09-11T08:54:27.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1674036224 bytes 2024-09-11T08:54:27.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1758837760 bytes 2024-09-11T08:54:27.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1796752384 bytes 2024-09-11T08:54:27.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1814903808 bytes 2024-09-11T08:54:27.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1901349888 bytes 2024-09-11T08:54:28.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2012431360 bytes 2024-09-11T08:54:28.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2039480320 bytes 2024-09-11T08:54:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2128999424 bytes 2024-09-11T08:54:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:54:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:54:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:54:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:28.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_21 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_21 snapshot 128M 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_21 128M 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:28.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:28.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_21 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_21 snapshot 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_21 snapshot 2024-09-11T08:54:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:54:29.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_21 2024-09-11T08:54:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_21': 2024-09-11T08:54:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:54:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:54:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1355f0cb917c 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1355f0cb917c 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:28 2024 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:28 2024 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:28 2024 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: ae0c9716-d47a-4a34-8cf9-e01df79ded5c 2024-09-11T08:54:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:54:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:54:29.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_21 2024-09-11T08:54:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T08:54:29.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:54:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T08:54:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:54:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_21 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:54:30.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 75211776 bytes 2024-09-11T08:54:30.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 98861056 bytes 2024-09-11T08:54:30.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 149699584 bytes 2024-09-11T08:54:30.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 201349120 bytes 2024-09-11T08:54:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 204762112 bytes 2024-09-11T08:54:30.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 238198784 bytes 2024-09-11T08:54:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 238812160 bytes 2024-09-11T08:54:31.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 278067200 bytes 2024-09-11T08:54:31.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 302950400 bytes 2024-09-11T08:54:31.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 316902400 bytes 2024-09-11T08:54:31.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 322382848 bytes 2024-09-11T08:54:31.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 328649728 bytes 2024-09-11T08:54:31.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 366055424 bytes 2024-09-11T08:54:31.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 427195392 bytes 2024-09-11T08:54:32.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 442375168 bytes 2024-09-11T08:54:32.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 460617728 bytes 2024-09-11T08:54:32.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 523691008 bytes 2024-09-11T08:54:32.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 586071040 bytes 2024-09-11T08:54:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 621117440 bytes 2024-09-11T08:54:33.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 689540096 bytes 2024-09-11T08:54:33.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 748857344 bytes 2024-09-11T08:54:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 782467072 bytes 2024-09-11T08:54:33.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 801462272 bytes 2024-09-11T08:54:33.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 842934272 bytes 2024-09-11T08:54:34.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 928963584 bytes 2024-09-11T08:54:34.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1008774144 bytes 2024-09-11T08:54:34.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1077425152 bytes 2024-09-11T08:54:34.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1131412480 bytes 2024-09-11T08:54:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1254168576 bytes 2024-09-11T08:54:35.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1298440192 bytes 2024-09-11T08:54:35.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1352401920 bytes 2024-09-11T08:54:35.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1371769856 bytes 2024-09-11T08:54:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:54:35.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_22 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_22 snapshot 128M 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T08:54:35.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_22 128M 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:54:35.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:54:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:35.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_22 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_22 snapshot 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_22 snapshot 2024-09-11T08:54:36.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:54:36.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_22 2024-09-11T08:54:36.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_22': 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13619b4b4398 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13619b4b4398 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:35 2024 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:35 2024 2024-09-11T08:54:36.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:35 2024 2024-09-11T08:54:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:54:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:54:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 9ceefcdb-b3e9-4601-8f51-a204c212530d 2024-09-11T08:54:36.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:54:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:54:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_22 2024-09-11T08:54:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T08:54:36.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:54:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=10 2024-09-11T08:54:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:54:36.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_22 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:54:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 47297536 bytes 2024-09-11T08:54:37.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 98390016 bytes 2024-09-11T08:54:37.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 127386624 bytes 2024-09-11T08:54:37.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 153893888 bytes 2024-09-11T08:54:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 285006848 bytes 2024-09-11T08:54:38.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 369523712 bytes 2024-09-11T08:54:38.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 379918336 bytes 2024-09-11T08:54:38.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 437540864 bytes 2024-09-11T08:54:38.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 495409152 bytes 2024-09-11T08:54:38.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 520981504 bytes 2024-09-11T08:54:39.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 624824320 bytes 2024-09-11T08:54:39.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 674802688 bytes 2024-09-11T08:54:39.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 747161600 bytes 2024-09-11T08:54:40.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 806491136 bytes 2024-09-11T08:54:40.264 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 890886144 bytes 2024-09-11T08:54:40.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 915062784 bytes 2024-09-11T08:54:40.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 943933440 bytes 2024-09-11T08:54:40.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 991583232 bytes 2024-09-11T08:54:41.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1078831104 bytes 2024-09-11T08:54:41.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1141807104 bytes 2024-09-11T08:54:41.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1191694336 bytes 2024-09-11T08:54:41.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1204491264 bytes 2024-09-11T08:54:41.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1228331008 bytes 2024-09-11T08:54:41.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1228953600 bytes 2024-09-11T08:54:42.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1275147264 bytes 2024-09-11T08:54:42.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1304991744 bytes 2024-09-11T08:54:42.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1323711488 bytes 2024-09-11T08:54:42.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1369858048 bytes 2024-09-11T08:54:42.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1440035840 bytes 2024-09-11T08:54:42.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1459683328 bytes 2024-09-11T08:54:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1483579392 bytes 2024-09-11T08:54:43.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1576091648 bytes 2024-09-11T08:54:43.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1621781504 bytes 2024-09-11T08:54:43.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1643613184 bytes 2024-09-11T08:54:43.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1683500032 bytes 2024-09-11T08:54:44.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1723555840 bytes 2024-09-11T08:54:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1748227072 bytes 2024-09-11T08:54:44.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1853518848 bytes 2024-09-11T08:54:44.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1882838016 bytes 2024-09-11T08:54:44.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1931049984 bytes 2024-09-11T08:54:45.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1990321152 bytes 2024-09-11T08:54:45.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 2035076096 bytes 2024-09-11T08:54:45.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 2066670592 bytes 2024-09-11T08:54:45.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 2104228864 bytes 2024-09-11T08:54:45.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2154245120 bytes 2024-09-11T08:54:45.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2181252096 bytes 2024-09-11T08:54:46.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2221568000 bytes 2024-09-11T08:54:46.268 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2262246400 bytes 2024-09-11T08:54:46.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2296128512 bytes 2024-09-11T08:54:46.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2331567104 bytes 2024-09-11T08:54:46.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_23 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_23 snapshot 128M 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_23 128M 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:54:46.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:54:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:46.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_23 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_23 snapshot 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T08:54:46.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:46.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_23 snapshot 2024-09-11T08:54:47.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:54:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_23 2024-09-11T08:54:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_23': 2024-09-11T08:54:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:54:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:54:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:54:47.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 136ddcc824d1 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.136ddcc824d1 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:46 2024 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:46 2024 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:46 2024 2024-09-11T08:54:47.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:54:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:54:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 69e0ab8a-356e-4e12-8363-b402fa05e07f 2024-09-11T08:54:47.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:54:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:54:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_23 2024-09-11T08:54:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:47.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T08:54:47.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:54:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:54:47.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:54:47.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_23 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:54:48.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 48458752 bytes 2024-09-11T08:54:48.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 68740096 bytes 2024-09-11T08:54:48.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 124051456 bytes 2024-09-11T08:54:48.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 215316480 bytes 2024-09-11T08:54:48.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 252974080 bytes 2024-09-11T08:54:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 299892736 bytes 2024-09-11T08:54:49.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 347887616 bytes 2024-09-11T08:54:49.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 374515712 bytes 2024-09-11T08:54:49.526 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 393397248 bytes 2024-09-11T08:54:49.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 423697408 bytes 2024-09-11T08:54:49.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 426671104 bytes 2024-09-11T08:54:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 431911936 bytes 2024-09-11T08:54:50.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 489488384 bytes 2024-09-11T08:54:50.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 508894208 bytes 2024-09-11T08:54:50.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 557865984 bytes 2024-09-11T08:54:50.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 675594240 bytes 2024-09-11T08:54:51.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 717371392 bytes 2024-09-11T08:54:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 728925184 bytes 2024-09-11T08:54:51.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 756425728 bytes 2024-09-11T08:54:51.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 799307776 bytes 2024-09-11T08:54:51.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 800953344 bytes 2024-09-11T08:54:52.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 847291392 bytes 2024-09-11T08:54:52.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 861330432 bytes 2024-09-11T08:54:52.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 906833920 bytes 2024-09-11T08:54:52.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 968930304 bytes 2024-09-11T08:54:52.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1007733760 bytes 2024-09-11T08:54:52.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1029872640 bytes 2024-09-11T08:54:53.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1063515136 bytes 2024-09-11T08:54:53.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1124074496 bytes 2024-09-11T08:54:53.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1213538304 bytes 2024-09-11T08:54:53.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1245539328 bytes 2024-09-11T08:54:53.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1253813248 bytes 2024-09-11T08:54:53.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1291201536 bytes 2024-09-11T08:54:54.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1306346496 bytes 2024-09-11T08:54:54.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1342405632 bytes 2024-09-11T08:54:54.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1387226112 bytes 2024-09-11T08:54:54.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1427829760 bytes 2024-09-11T08:54:54.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1461303296 bytes 2024-09-11T08:54:54.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1477160960 bytes 2024-09-11T08:54:55.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1484103680 bytes 2024-09-11T08:54:55.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1484717056 bytes 2024-09-11T08:54:55.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1516800000 bytes 2024-09-11T08:54:55.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1577313280 bytes 2024-09-11T08:54:55.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1660918784 bytes 2024-09-11T08:54:55.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1717755904 bytes 2024-09-11T08:54:56.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1808220160 bytes 2024-09-11T08:54:56.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1845994496 bytes 2024-09-11T08:54:56.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1905634304 bytes 2024-09-11T08:54:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:54:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:54:56.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:54:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:54:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_24 2024-09-11T08:54:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_24 snapshot 128M 2024-09-11T08:54:56.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T08:54:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:54:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_24 128M 2024-09-11T08:54:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T08:54:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:54:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:54:56.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:54:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:54:56.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_24 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_24 snapshot 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T08:54:56.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:54:56.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_24 snapshot 2024-09-11T08:54:57.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:54:57.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_24 2024-09-11T08:54:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_24': 2024-09-11T08:54:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:54:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:54:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:54:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1379da599895 2024-09-11T08:54:57.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1379da599895 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:56 2024 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:56 2024 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:56 2024 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 523e3106-9d30-48a3-8d7d-0a45f9581e5e 2024-09-11T08:54:57.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:54:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:54:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_24 2024-09-11T08:54:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:54:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:54:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T08:54:57.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:54:57.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_24 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:54:58.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 605184 bytes 2024-09-11T08:54:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 35119104 bytes 2024-09-11T08:54:58.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 67522560 bytes 2024-09-11T08:54:58.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 76635136 bytes 2024-09-11T08:54:58.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 141133824 bytes 2024-09-11T08:54:58.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 198684672 bytes 2024-09-11T08:54:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 252991488 bytes 2024-09-11T08:54:59.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 314192896 bytes 2024-09-11T08:54:59.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 382269440 bytes 2024-09-11T08:54:59.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 408945664 bytes 2024-09-11T08:54:59.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 474184704 bytes 2024-09-11T08:55:00.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 496475136 bytes 2024-09-11T08:55:00.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 518042624 bytes 2024-09-11T08:55:00.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 554885120 bytes 2024-09-11T08:55:00.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 579067904 bytes 2024-09-11T08:55:00.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 592397312 bytes 2024-09-11T08:55:00.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 680582144 bytes 2024-09-11T08:55:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 718110720 bytes 2024-09-11T08:55:01.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 742475776 bytes 2024-09-11T08:55:01.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 789173248 bytes 2024-09-11T08:55:01.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 821344256 bytes 2024-09-11T08:55:01.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 842331136 bytes 2024-09-11T08:55:01.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 880429056 bytes 2024-09-11T08:55:02.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 922717184 bytes 2024-09-11T08:55:02.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1000311808 bytes 2024-09-11T08:55:02.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1018355712 bytes 2024-09-11T08:55:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1087547392 bytes 2024-09-11T08:55:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1134620672 bytes 2024-09-11T08:55:02.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1140411392 bytes 2024-09-11T08:55:03.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1160157184 bytes 2024-09-11T08:55:03.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1213851648 bytes 2024-09-11T08:55:03.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1280498688 bytes 2024-09-11T08:55:03.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1292657664 bytes 2024-09-11T08:55:04.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1328791552 bytes 2024-09-11T08:55:04.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1348651008 bytes 2024-09-11T08:55:04.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1368804352 bytes 2024-09-11T08:55:04.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1416891392 bytes 2024-09-11T08:55:04.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1433218048 bytes 2024-09-11T08:55:04.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1510833152 bytes 2024-09-11T08:55:05.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1584755712 bytes 2024-09-11T08:55:05.258 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T08:55:05.259 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T08:55:05.259 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-11T08:55:05.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T08:55:05.264+0000 7fa68fa85640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 36198) UID: 1000 2024-09-11T08:55:05.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1647357952 bytes 2024-09-11T08:55:05.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1687787520 bytes 2024-09-11T08:55:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1843451904 bytes 2024-09-11T08:55:06.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1871544320 bytes 2024-09-11T08:55:06.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1943630848 bytes 2024-09-11T08:55:06.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1954378752 bytes 2024-09-11T08:55:06.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2059619328 bytes 2024-09-11T08:55:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:55:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:55:06.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_25 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_25 snapshot 128M 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_25 128M 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:55:06.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:55:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:06.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_25 2024-09-11T08:55:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:55:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:55:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_25 snapshot 2024-09-11T08:55:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:06.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T08:55:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:06.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_25 snapshot 2024-09-11T08:55:07.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:55:07.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_25 2024-09-11T08:55:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_25': 2024-09-11T08:55:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:55:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:55:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:55:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1385cecdba67 2024-09-11T08:55:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1385cecdba67 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:06 2024 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:06 2024 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:06 2024 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 0534cedd-e737-4e3d-87e8-aff6ea511c46 2024-09-11T08:55:07.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:55:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:55:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_25 2024-09-11T08:55:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:07.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T08:55:07.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:55:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T08:55:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:55:07.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_25 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:55:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 27855872 bytes 2024-09-11T08:55:07.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 78972928 bytes 2024-09-11T08:55:08.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 127582208 bytes 2024-09-11T08:55:08.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 151094272 bytes 2024-09-11T08:55:08.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 239099904 bytes 2024-09-11T08:55:08.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 246326272 bytes 2024-09-11T08:55:08.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 322649088 bytes 2024-09-11T08:55:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 378252288 bytes 2024-09-11T08:55:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 497389568 bytes 2024-09-11T08:55:09.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 528468992 bytes 2024-09-11T08:55:09.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 567142400 bytes 2024-09-11T08:55:09.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 595802112 bytes 2024-09-11T08:55:09.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 624200704 bytes 2024-09-11T08:55:10.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 657022976 bytes 2024-09-11T08:55:10.359 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 704022528 bytes 2024-09-11T08:55:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 790324224 bytes 2024-09-11T08:55:10.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 810312704 bytes 2024-09-11T08:55:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 831716352 bytes 2024-09-11T08:55:11.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 872350720 bytes 2024-09-11T08:55:11.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 980236288 bytes 2024-09-11T08:55:11.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1008950272 bytes 2024-09-11T08:55:11.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1015784448 bytes 2024-09-11T08:55:11.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1056221184 bytes 2024-09-11T08:55:11.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1069112320 bytes 2024-09-11T08:55:12.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1094674432 bytes 2024-09-11T08:55:12.260 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T08:55:12.260 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:55:12.261 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T08:55:12.261 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T08:55:12.261 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T08:55:12.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T08:55:12.264 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-11T08:55:12.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1174597632 bytes 2024-09-11T08:55:12.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1222324224 bytes 2024-09-11T08:55:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1291353088 bytes 2024-09-11T08:55:12.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1291980800 bytes 2024-09-11T08:55:13.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1337200640 bytes 2024-09-11T08:55:13.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1377771520 bytes 2024-09-11T08:55:13.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1378397184 bytes 2024-09-11T08:55:13.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_26 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_26 snapshot 128M 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:55:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_26 128M 2024-09-11T08:55:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:13.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:13.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_26 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_26 snapshot 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:13.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T08:55:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:13.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_26 snapshot 2024-09-11T08:55:14.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:55:14.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_26 2024-09-11T08:55:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_26': 2024-09-11T08:55:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:55:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:55:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:55:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 139bb034ef35 2024-09-11T08:55:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.139bb034ef35 2024-09-11T08:55:14.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:13 2024 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:13 2024 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:13 2024 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: de26d284-def3-43fc-8308-3c61d65c9c5f 2024-09-11T08:55:14.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:55:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:55:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_26 2024-09-11T08:55:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T08:55:14.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:55:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=11 2024-09-11T08:55:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:55:14.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_26 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:55:14.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 64144384 bytes 2024-09-11T08:55:15.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 93633536 bytes 2024-09-11T08:55:15.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 122148864 bytes 2024-09-11T08:55:15.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 122754048 bytes 2024-09-11T08:55:15.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 171598848 bytes 2024-09-11T08:55:15.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 243121152 bytes 2024-09-11T08:55:16.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 325529600 bytes 2024-09-11T08:55:16.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 367548416 bytes 2024-09-11T08:55:16.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 386419712 bytes 2024-09-11T08:55:16.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 412694528 bytes 2024-09-11T08:55:16.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 416849920 bytes 2024-09-11T08:55:16.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 459558912 bytes 2024-09-11T08:55:16.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 485180416 bytes 2024-09-11T08:55:17.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 539855872 bytes 2024-09-11T08:55:17.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 575347712 bytes 2024-09-11T08:55:17.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 605765632 bytes 2024-09-11T08:55:17.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 635071488 bytes 2024-09-11T08:55:17.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 663910400 bytes 2024-09-11T08:55:17.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 688436224 bytes 2024-09-11T08:55:18.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 727229440 bytes 2024-09-11T08:55:18.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 730987520 bytes 2024-09-11T08:55:18.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 783286272 bytes 2024-09-11T08:55:18.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 889398272 bytes 2024-09-11T08:55:18.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 956827648 bytes 2024-09-11T08:55:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1052451840 bytes 2024-09-11T08:55:19.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1080806400 bytes 2024-09-11T08:55:19.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1118583808 bytes 2024-09-11T08:55:19.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1136722944 bytes 2024-09-11T08:55:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1159012352 bytes 2024-09-11T08:55:20.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1190801408 bytes 2024-09-11T08:55:20.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1207476224 bytes 2024-09-11T08:55:20.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1230694400 bytes 2024-09-11T08:55:20.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1279246336 bytes 2024-09-11T08:55:20.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1279859712 bytes 2024-09-11T08:55:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1384812544 bytes 2024-09-11T08:55:21.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1454292992 bytes 2024-09-11T08:55:21.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1474846720 bytes 2024-09-11T08:55:21.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1489817600 bytes 2024-09-11T08:55:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1513129984 bytes 2024-09-11T08:55:21.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1557957632 bytes 2024-09-11T08:55:21.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1615849472 bytes 2024-09-11T08:55:22.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1644130304 bytes 2024-09-11T08:55:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1683363840 bytes 2024-09-11T08:55:22.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1723943936 bytes 2024-09-11T08:55:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1724565504 bytes 2024-09-11T08:55:22.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1783989248 bytes 2024-09-11T08:55:22.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1785598976 bytes 2024-09-11T08:55:23.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1833078784 bytes 2024-09-11T08:55:23.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1858114560 bytes 2024-09-11T08:55:23.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1968621568 bytes 2024-09-11T08:55:23.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2021473280 bytes 2024-09-11T08:55:23.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2046215168 bytes 2024-09-11T08:55:24.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2097529856 bytes 2024-09-11T08:55:24.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2164385792 bytes 2024-09-11T08:55:24.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2237303808 bytes 2024-09-11T08:55:24.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2314897408 bytes 2024-09-11T08:55:25.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2355918848 bytes 2024-09-11T08:55:25.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2393229312 bytes 2024-09-11T08:55:25.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2465129472 bytes 2024-09-11T08:55:25.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2512450560 bytes 2024-09-11T08:55:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:55:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:55:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:55:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:25.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_27 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_27 snapshot 128M 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_27 128M 2024-09-11T08:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T08:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:55:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:55:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:55:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:25.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_27 2024-09-11T08:55:25.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_27 snapshot 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:25.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_27 snapshot 2024-09-11T08:55:26.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:55:26.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_27 2024-09-11T08:55:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_27': 2024-09-11T08:55:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:55:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:55:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:55:26.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13a754011a13 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13a754011a13 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:25 2024 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:25 2024 2024-09-11T08:55:26.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:25 2024 2024-09-11T08:55:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:55:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:55:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8c3ff342-3906-447a-9223-131a82f747f2 2024-09-11T08:55:26.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:55:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:55:26.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_27 2024-09-11T08:55:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T08:55:26.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:55:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=8 2024-09-11T08:55:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:55:26.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_27 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:55:26.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 30214144 bytes 2024-09-11T08:55:27.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 88914944 bytes 2024-09-11T08:55:27.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 135267328 bytes 2024-09-11T08:55:27.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 177925120 bytes 2024-09-11T08:55:27.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 221085696 bytes 2024-09-11T08:55:27.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 258985984 bytes 2024-09-11T08:55:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 288699392 bytes 2024-09-11T08:55:28.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 320130048 bytes 2024-09-11T08:55:28.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 344146944 bytes 2024-09-11T08:55:28.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 378826752 bytes 2024-09-11T08:55:28.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 384698368 bytes 2024-09-11T08:55:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 393967616 bytes 2024-09-11T08:55:28.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 406274048 bytes 2024-09-11T08:55:29.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 461260800 bytes 2024-09-11T08:55:29.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 491388928 bytes 2024-09-11T08:55:29.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 523985920 bytes 2024-09-11T08:55:29.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 524585984 bytes 2024-09-11T08:55:29.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 595198976 bytes 2024-09-11T08:55:30.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 695343104 bytes 2024-09-11T08:55:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 702507008 bytes 2024-09-11T08:55:30.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 703128576 bytes 2024-09-11T08:55:30.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 730662912 bytes 2024-09-11T08:55:30.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 779401216 bytes 2024-09-11T08:55:30.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 829023232 bytes 2024-09-11T08:55:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 830619648 bytes 2024-09-11T08:55:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 864126976 bytes 2024-09-11T08:55:31.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 954489856 bytes 2024-09-11T08:55:31.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 982819840 bytes 2024-09-11T08:55:31.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1051148288 bytes 2024-09-11T08:55:31.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1070976000 bytes 2024-09-11T08:55:32.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1115931648 bytes 2024-09-11T08:55:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1177298944 bytes 2024-09-11T08:55:32.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1261700096 bytes 2024-09-11T08:55:32.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1272069120 bytes 2024-09-11T08:55:32.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1285691392 bytes 2024-09-11T08:55:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1359696896 bytes 2024-09-11T08:55:33.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1429594112 bytes 2024-09-11T08:55:33.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1453864960 bytes 2024-09-11T08:55:33.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1476660224 bytes 2024-09-11T08:55:33.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1528519680 bytes 2024-09-11T08:55:34.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1548868608 bytes 2024-09-11T08:55:34.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1599012864 bytes 2024-09-11T08:55:34.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1636930560 bytes 2024-09-11T08:55:34.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1705404416 bytes 2024-09-11T08:55:34.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_28 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_28 snapshot 128M 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T08:55:34.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_28 128M 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:55:34.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:55:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:34.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_28 2024-09-11T08:55:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:55:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:55:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_28 snapshot 2024-09-11T08:55:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:34.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T08:55:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:34.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_28 snapshot 2024-09-11T08:55:35.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:55:35.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_28 2024-09-11T08:55:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_28': 2024-09-11T08:55:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:55:35.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13b3fdda34d1 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13b3fdda34d1 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:34 2024 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:34 2024 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:34 2024 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:55:35.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: fece848e-5f36-407c-83b5-17909bfc414d 2024-09-11T08:55:35.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:55:35.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:55:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_28 2024-09-11T08:55:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T08:55:35.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:55:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:55:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:55:35.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_28 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:55:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 85730304 bytes 2024-09-11T08:55:36.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 112291840 bytes 2024-09-11T08:55:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 180916224 bytes 2024-09-11T08:55:36.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 202455040 bytes 2024-09-11T08:55:36.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 207329280 bytes 2024-09-11T08:55:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 272345088 bytes 2024-09-11T08:55:37.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 286170112 bytes 2024-09-11T08:55:37.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 339187712 bytes 2024-09-11T08:55:37.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 373760000 bytes 2024-09-11T08:55:37.525 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 391094272 bytes 2024-09-11T08:55:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 398973952 bytes 2024-09-11T08:55:37.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 447917056 bytes 2024-09-11T08:55:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 495922176 bytes 2024-09-11T08:55:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 526627840 bytes 2024-09-11T08:55:38.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 561293312 bytes 2024-09-11T08:55:38.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 579346432 bytes 2024-09-11T08:55:38.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 628335616 bytes 2024-09-11T08:55:38.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 656524288 bytes 2024-09-11T08:55:39.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 743682048 bytes 2024-09-11T08:55:39.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 799095808 bytes 2024-09-11T08:55:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 862330880 bytes 2024-09-11T08:55:39.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 959895552 bytes 2024-09-11T08:55:39.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1002545152 bytes 2024-09-11T08:55:40.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1045918720 bytes 2024-09-11T08:55:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1144352768 bytes 2024-09-11T08:55:40.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1210170368 bytes 2024-09-11T08:55:40.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1280946176 bytes 2024-09-11T08:55:41.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1362034688 bytes 2024-09-11T08:55:41.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1385552896 bytes 2024-09-11T08:55:41.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1457324032 bytes 2024-09-11T08:55:41.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1502889984 bytes 2024-09-11T08:55:41.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1526136832 bytes 2024-09-11T08:55:42.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1550959616 bytes 2024-09-11T08:55:42.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1569037312 bytes 2024-09-11T08:55:42.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1639967744 bytes 2024-09-11T08:55:42.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1717301248 bytes 2024-09-11T08:55:42.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1785162752 bytes 2024-09-11T08:55:43.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1818199040 bytes 2024-09-11T08:55:43.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1836161024 bytes 2024-09-11T08:55:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1877464064 bytes 2024-09-11T08:55:43.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1913745408 bytes 2024-09-11T08:55:43.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1943003136 bytes 2024-09-11T08:55:43.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1977655296 bytes 2024-09-11T08:55:44.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 2049931264 bytes 2024-09-11T08:55:44.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2071542784 bytes 2024-09-11T08:55:44.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2112528384 bytes 2024-09-11T08:55:44.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2141732864 bytes 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_29 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_29 snapshot 128M 2024-09-11T08:55:44.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_29 128M 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:44.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_29 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_29 snapshot 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:44.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_29 snapshot 2024-09-11T08:55:45.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:55:45.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_29 2024-09-11T08:55:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_29': 2024-09-11T08:55:45.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13bfa6a002a9 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13bfa6a002a9 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:44 2024 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:44 2024 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:44 2024 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:55:45.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 2a587c46-253d-496d-be6f-0d9d4063df36 2024-09-11T08:55:45.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:55:45.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:55:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_29 2024-09-11T08:55:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T08:55:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:55:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=14 2024-09-11T08:55:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:55:45.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_29 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:55:45.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 22243328 bytes 2024-09-11T08:55:46.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 46629888 bytes 2024-09-11T08:55:46.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 136701952 bytes 2024-09-11T08:55:46.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 156579840 bytes 2024-09-11T08:55:46.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 177426432 bytes 2024-09-11T08:55:46.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 228719616 bytes 2024-09-11T08:55:47.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 281712640 bytes 2024-09-11T08:55:47.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 363063296 bytes 2024-09-11T08:55:47.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 394574848 bytes 2024-09-11T08:55:47.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 443162624 bytes 2024-09-11T08:55:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 478005248 bytes 2024-09-11T08:55:48.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 515206144 bytes 2024-09-11T08:55:48.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 549551104 bytes 2024-09-11T08:55:48.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 575855616 bytes 2024-09-11T08:55:48.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 604377088 bytes 2024-09-11T08:55:48.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 639621120 bytes 2024-09-11T08:55:48.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 655015936 bytes 2024-09-11T08:55:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 675148800 bytes 2024-09-11T08:55:49.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 686810112 bytes 2024-09-11T08:55:49.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 712151040 bytes 2024-09-11T08:55:49.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 743497728 bytes 2024-09-11T08:55:49.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 762058752 bytes 2024-09-11T08:55:49.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 798430208 bytes 2024-09-11T08:55:49.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 830309376 bytes 2024-09-11T08:55:50.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 865050624 bytes 2024-09-11T08:55:50.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 875414528 bytes 2024-09-11T08:55:50.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 914869248 bytes 2024-09-11T08:55:50.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 959295488 bytes 2024-09-11T08:55:50.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 983366656 bytes 2024-09-11T08:55:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1024749568 bytes 2024-09-11T08:55:51.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1071348736 bytes 2024-09-11T08:55:51.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1154707456 bytes 2024-09-11T08:55:51.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1216976896 bytes 2024-09-11T08:55:51.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1232420864 bytes 2024-09-11T08:55:51.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1250084864 bytes 2024-09-11T08:55:52.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1324511232 bytes 2024-09-11T08:55:52.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1377224704 bytes 2024-09-11T08:55:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1445690368 bytes 2024-09-11T08:55:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1522540544 bytes 2024-09-11T08:55:52.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1545488384 bytes 2024-09-11T08:55:53.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1557380096 bytes 2024-09-11T08:55:53.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1640194048 bytes 2024-09-11T08:55:53.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1692585984 bytes 2024-09-11T08:55:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1713960960 bytes 2024-09-11T08:55:53.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1736671232 bytes 2024-09-11T08:55:54.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1819655168 bytes 2024-09-11T08:55:54.270 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1846821888 bytes 2024-09-11T08:55:54.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1905276928 bytes 2024-09-11T08:55:54.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1958773760 bytes 2024-09-11T08:55:54.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2020415488 bytes 2024-09-11T08:55:55.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2074721280 bytes 2024-09-11T08:55:55.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2098633728 bytes 2024-09-11T08:55:55.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2136478720 bytes 2024-09-11T08:55:55.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2152903680 bytes 2024-09-11T08:55:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2209235968 bytes 2024-09-11T08:55:55.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2234556416 bytes 2024-09-11T08:55:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2280018944 bytes 2024-09-11T08:55:56.370 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.711770287061747) > weight (0.25) 2024-09-11T08:55:56.370 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.8712805248313042) > weight (0.25) 2024-09-11T08:55:56.371 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4313911253238111) > weight (0.25) 2024-09-11T08:55:56.371 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.3 2024-09-11T08:55:56.371 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Sent signal 15 2024-09-11T08:55:56.371 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T08:55:56.372 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T08:55:56.375+0000 7ff64ef32640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 30824) UID: 1000 2024-09-11T08:55:56.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2353654784 bytes 2024-09-11T08:55:56.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2424810496 bytes 2024-09-11T08:55:56.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2448434176 bytes 2024-09-11T08:55:56.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2514163712 bytes 2024-09-11T08:55:57.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2514769920 bytes 2024-09-11T08:55:57.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2552651776 bytes 2024-09-11T08:55:57.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2564810752 bytes 2024-09-11T08:55:57.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2635280384 bytes 2024-09-11T08:55:57.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2663456768 bytes 2024-09-11T08:55:57.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2692674560 bytes 2024-09-11T08:55:58.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2718441472 bytes 2024-09-11T08:55:58.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2734860288 bytes 2024-09-11T08:55:58.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2832601088 bytes 2024-09-11T08:55:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2833193984 bytes 2024-09-11T08:55:58.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2860584960 bytes 2024-09-11T08:55:58.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 2880248832 bytes 2024-09-11T08:55:59.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 2941033472 bytes 2024-09-11T08:55:59.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 2988831744 bytes 2024-09-11T08:55:59.457 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 3017423872 bytes 2024-09-11T08:55:59.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7700 IOs, 3064078336 bytes 2024-09-11T08:55:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:55:59.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_30 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_30 snapshot 128M 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:59.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_30 128M 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:55:59.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_30 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_30 snapshot 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:55:59.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_30 snapshot 2024-09-11T08:56:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:56:00.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_30 2024-09-11T08:56:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_30': 2024-09-11T08:56:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:56:00.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13c9d246eda1 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13c9d246eda1 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:59 2024 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:59 2024 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:59 2024 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:56:00.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:56:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: eb74963f-2883-4117-8704-7dd16bb13e33 2024-09-11T08:56:00.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:56:00.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:56:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_30 2024-09-11T08:56:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T08:56:00.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:56:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T08:56:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:56:00.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_30 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:56:01.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 22196224 bytes 2024-09-11T08:56:01.265 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 50262016 bytes 2024-09-11T08:56:01.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 138694656 bytes 2024-09-11T08:56:01.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 193586176 bytes 2024-09-11T08:56:01.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 246975488 bytes 2024-09-11T08:56:02.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 247589888 bytes 2024-09-11T08:56:02.268 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 316632064 bytes 2024-09-11T08:56:02.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 376213504 bytes 2024-09-11T08:56:02.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 376824832 bytes 2024-09-11T08:56:02.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 414524416 bytes 2024-09-11T08:56:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 445011968 bytes 2024-09-11T08:56:03.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 476221440 bytes 2024-09-11T08:56:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 543014912 bytes 2024-09-11T08:56:03.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 559667200 bytes 2024-09-11T08:56:03.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 587808768 bytes 2024-09-11T08:56:03.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 668961792 bytes 2024-09-11T08:56:03.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 723251200 bytes 2024-09-11T08:56:04.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 808217600 bytes 2024-09-11T08:56:04.372 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.3 2024-09-11T08:56:04.372 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:56:04.373 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.3 2024-09-11T08:56:04.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Restarting daemon 2024-09-11T08:56:04.374 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 2024-09-11T08:56:04.377 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Started 2024-09-11T08:56:04.377 INFO:tasks.rbd_mirror_thrash:waiting for 72 secs before thrashing 2024-09-11T08:56:04.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 808841216 bytes 2024-09-11T08:56:04.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 855312384 bytes 2024-09-11T08:56:04.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 923506688 bytes 2024-09-11T08:56:05.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 962998272 bytes 2024-09-11T08:56:05.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 985236480 bytes 2024-09-11T08:56:05.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1004911616 bytes 2024-09-11T08:56:05.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1015859200 bytes 2024-09-11T08:56:05.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1055443968 bytes 2024-09-11T08:56:05.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1087924224 bytes 2024-09-11T08:56:05.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1135221760 bytes 2024-09-11T08:56:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1166510080 bytes 2024-09-11T08:56:06.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1206863872 bytes 2024-09-11T08:56:06.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1321530368 bytes 2024-09-11T08:56:06.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1355248640 bytes 2024-09-11T08:56:06.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1359112192 bytes 2024-09-11T08:56:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:56:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:56:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:56:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_31 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_31 snapshot 128M 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_31 128M 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:06.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_31 2024-09-11T08:56:07.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_31 snapshot 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_31 snapshot 2024-09-11T08:56:07.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:56:07.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_31 2024-09-11T08:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_31': 2024-09-11T08:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13d8f04dcc8c 2024-09-11T08:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13d8f04dcc8c 2024-09-11T08:56:07.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:07 2024 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:07 2024 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:07 2024 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a8e3428e-5e9d-4305-a12d-e6118689e825 2024-09-11T08:56:07.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:56:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:56:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_31 2024-09-11T08:56:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:07.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T08:56:07.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=10 2024-09-11T08:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:56:07.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_31 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:56:07.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 2341888 bytes 2024-09-11T08:56:07.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 71856128 bytes 2024-09-11T08:56:08.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 105893888 bytes 2024-09-11T08:56:08.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 152283136 bytes 2024-09-11T08:56:08.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 185188352 bytes 2024-09-11T08:56:08.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 212734976 bytes 2024-09-11T08:56:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 262837248 bytes 2024-09-11T08:56:09.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 339298304 bytes 2024-09-11T08:56:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 339928064 bytes 2024-09-11T08:56:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 363459584 bytes 2024-09-11T08:56:09.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 408171520 bytes 2024-09-11T08:56:09.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 482571264 bytes 2024-09-11T08:56:09.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 494653440 bytes 2024-09-11T08:56:10.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 528050176 bytes 2024-09-11T08:56:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 574979072 bytes 2024-09-11T08:56:10.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 633857024 bytes 2024-09-11T08:56:10.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 656591872 bytes 2024-09-11T08:56:10.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 734841856 bytes 2024-09-11T08:56:11.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 744845312 bytes 2024-09-11T08:56:11.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 781330432 bytes 2024-09-11T08:56:11.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 788970496 bytes 2024-09-11T08:56:11.524 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 841393152 bytes 2024-09-11T08:56:11.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 905777152 bytes 2024-09-11T08:56:11.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 979855360 bytes 2024-09-11T08:56:12.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1005819904 bytes 2024-09-11T08:56:12.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1030073344 bytes 2024-09-11T08:56:12.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1083503616 bytes 2024-09-11T08:56:12.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1116018688 bytes 2024-09-11T08:56:12.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1140154368 bytes 2024-09-11T08:56:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1212376064 bytes 2024-09-11T08:56:13.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1212959744 bytes 2024-09-11T08:56:13.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1259522048 bytes 2024-09-11T08:56:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1260239872 bytes 2024-09-11T08:56:13.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1332282368 bytes 2024-09-11T08:56:13.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1381992448 bytes 2024-09-11T08:56:14.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1439696896 bytes 2024-09-11T08:56:14.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1519825920 bytes 2024-09-11T08:56:14.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1597881344 bytes 2024-09-11T08:56:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1621174272 bytes 2024-09-11T08:56:15.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1688173568 bytes 2024-09-11T08:56:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1725968384 bytes 2024-09-11T08:56:15.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1747289088 bytes 2024-09-11T08:56:15.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1748078592 bytes 2024-09-11T08:56:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1809614848 bytes 2024-09-11T08:56:15.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1815152640 bytes 2024-09-11T08:56:15.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1836174336 bytes 2024-09-11T08:56:16.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1863840768 bytes 2024-09-11T08:56:16.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1885522944 bytes 2024-09-11T08:56:16.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1915241472 bytes 2024-09-11T08:56:16.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1954768896 bytes 2024-09-11T08:56:16.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 1971912704 bytes 2024-09-11T08:56:16.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 1990761472 bytes 2024-09-11T08:56:17.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2054226944 bytes 2024-09-11T08:56:17.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2140211200 bytes 2024-09-11T08:56:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2143130624 bytes 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_32 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_32 snapshot 128M 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:17.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_32 128M 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T08:56:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:56:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:56:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:56:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:17.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_32 2024-09-11T08:56:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:56:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:56:17.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_32 snapshot 2024-09-11T08:56:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T08:56:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:17.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_32 snapshot 2024-09-11T08:56:18.265 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.3160582707128273) > weight (0.25) 2024-09-11T08:56:18.265 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T08:56:18.265 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T08:56:18.265 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-11T08:56:18.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T08:56:18.267+0000 7fb50a1a8640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 38901) UID: 1000 2024-09-11T08:56:18.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:56:18.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_32 2024-09-11T08:56:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_32': 2024-09-11T08:56:18.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13e12da85094 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13e12da85094 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:17 2024 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:17 2024 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:17 2024 2024-09-11T08:56:18.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:56:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:56:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 62829279-46f0-42ff-8ae0-7092ec6a74d0 2024-09-11T08:56:18.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:56:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:56:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_32 2024-09-11T08:56:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:18.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T08:56:18.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:56:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:56:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:56:18.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_32 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:56:19.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 104118272 bytes 2024-09-11T08:56:19.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 122200064 bytes 2024-09-11T08:56:19.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 168361984 bytes 2024-09-11T08:56:19.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 183493632 bytes 2024-09-11T08:56:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 213768192 bytes 2024-09-11T08:56:20.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 279398400 bytes 2024-09-11T08:56:20.266 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T08:56:20.266 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:56:20.267 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T08:56:20.267 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T08:56:20.267 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T08:56:20.269 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T08:56:20.269 INFO:tasks.rbd_mirror_thrash:waiting for 67 secs before thrashing 2024-09-11T08:56:20.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 411303936 bytes 2024-09-11T08:56:20.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 433764352 bytes 2024-09-11T08:56:20.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 520873984 bytes 2024-09-11T08:56:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 581806080 bytes 2024-09-11T08:56:21.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 605522944 bytes 2024-09-11T08:56:21.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 620960768 bytes 2024-09-11T08:56:21.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 677945344 bytes 2024-09-11T08:56:21.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 696642560 bytes 2024-09-11T08:56:21.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 727950336 bytes 2024-09-11T08:56:21.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 760757248 bytes 2024-09-11T08:56:22.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 781110272 bytes 2024-09-11T08:56:22.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 781718528 bytes 2024-09-11T08:56:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 822545408 bytes 2024-09-11T08:56:22.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 848245760 bytes 2024-09-11T08:56:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 860224512 bytes 2024-09-11T08:56:22.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 912948224 bytes 2024-09-11T08:56:22.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 945638400 bytes 2024-09-11T08:56:23.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1012622336 bytes 2024-09-11T08:56:23.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1070788608 bytes 2024-09-11T08:56:23.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1106795520 bytes 2024-09-11T08:56:23.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1115863040 bytes 2024-09-11T08:56:23.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1166357504 bytes 2024-09-11T08:56:24.121 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1231179776 bytes 2024-09-11T08:56:24.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1231784960 bytes 2024-09-11T08:56:24.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1243470848 bytes 2024-09-11T08:56:24.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1298061312 bytes 2024-09-11T08:56:24.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1372725248 bytes 2024-09-11T08:56:25.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1436674048 bytes 2024-09-11T08:56:25.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1465196544 bytes 2024-09-11T08:56:25.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1517292544 bytes 2024-09-11T08:56:25.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1553134592 bytes 2024-09-11T08:56:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:56:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:56:25.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_33 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_33 snapshot 128M 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_33 128M 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:56:25.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:56:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:25.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_33 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_33 snapshot 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:25.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_33 snapshot 2024-09-11T08:56:26.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:56:26.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_33 2024-09-11T08:56:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_33': 2024-09-11T08:56:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:56:26.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13f6fc293480 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13f6fc293480 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:56:26.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:25 2024 2024-09-11T08:56:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:25 2024 2024-09-11T08:56:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:25 2024 2024-09-11T08:56:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:56:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:56:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 3b06c806-6a3f-42c2-bb4e-e1f16e05e5d7 2024-09-11T08:56:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:56:26.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:56:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_33 2024-09-11T08:56:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T08:56:26.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:56:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=14 2024-09-11T08:56:26.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:56:26.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_33 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:56:26.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 32395264 bytes 2024-09-11T08:56:27.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 49475584 bytes 2024-09-11T08:56:27.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 124051456 bytes 2024-09-11T08:56:27.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 183539712 bytes 2024-09-11T08:56:27.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 196167680 bytes 2024-09-11T08:56:27.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 216028160 bytes 2024-09-11T08:56:27.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 222511104 bytes 2024-09-11T08:56:28.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 296737792 bytes 2024-09-11T08:56:28.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 351197184 bytes 2024-09-11T08:56:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 400355328 bytes 2024-09-11T08:56:28.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 429559808 bytes 2024-09-11T08:56:28.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 430150656 bytes 2024-09-11T08:56:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 508686336 bytes 2024-09-11T08:56:29.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 537742336 bytes 2024-09-11T08:56:29.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 609031168 bytes 2024-09-11T08:56:29.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 630083584 bytes 2024-09-11T08:56:29.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 706523136 bytes 2024-09-11T08:56:30.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 752520192 bytes 2024-09-11T08:56:30.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 804928512 bytes 2024-09-11T08:56:30.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 828538880 bytes 2024-09-11T08:56:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 858681344 bytes 2024-09-11T08:56:30.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 859285504 bytes 2024-09-11T08:56:30.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 884907008 bytes 2024-09-11T08:56:31.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 923106304 bytes 2024-09-11T08:56:31.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 967236608 bytes 2024-09-11T08:56:31.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1003631616 bytes 2024-09-11T08:56:31.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1029004288 bytes 2024-09-11T08:56:31.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1066379264 bytes 2024-09-11T08:56:31.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1090883584 bytes 2024-09-11T08:56:32.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1183479808 bytes 2024-09-11T08:56:32.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1214034944 bytes 2024-09-11T08:56:32.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1297699840 bytes 2024-09-11T08:56:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1353095168 bytes 2024-09-11T08:56:33.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1419674624 bytes 2024-09-11T08:56:33.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1463986176 bytes 2024-09-11T08:56:33.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1493095424 bytes 2024-09-11T08:56:33.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1547909120 bytes 2024-09-11T08:56:33.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1570290688 bytes 2024-09-11T08:56:34.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1610348544 bytes 2024-09-11T08:56:34.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1670147072 bytes 2024-09-11T08:56:34.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1727485952 bytes 2024-09-11T08:56:34.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1741263872 bytes 2024-09-11T08:56:34.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1801120768 bytes 2024-09-11T08:56:35.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1819928576 bytes 2024-09-11T08:56:35.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1824502784 bytes 2024-09-11T08:56:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1869441024 bytes 2024-09-11T08:56:35.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1930123264 bytes 2024-09-11T08:56:35.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2012814336 bytes 2024-09-11T08:56:36.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2064806912 bytes 2024-09-11T08:56:36.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2123026432 bytes 2024-09-11T08:56:36.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2164316160 bytes 2024-09-11T08:56:36.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2201653248 bytes 2024-09-11T08:56:36.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2226750464 bytes 2024-09-11T08:56:36.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2249657344 bytes 2024-09-11T08:56:36.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2250262528 bytes 2024-09-11T08:56:37.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2319748096 bytes 2024-09-11T08:56:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2387155968 bytes 2024-09-11T08:56:37.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2432121856 bytes 2024-09-11T08:56:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2457059328 bytes 2024-09-11T08:56:37.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2508042240 bytes 2024-09-11T08:56:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2528651264 bytes 2024-09-11T08:56:38.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2596460544 bytes 2024-09-11T08:56:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2619447296 bytes 2024-09-11T08:56:38.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2698001408 bytes 2024-09-11T08:56:38.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2760190976 bytes 2024-09-11T08:56:39.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2794823680 bytes 2024-09-11T08:56:39.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2828406784 bytes 2024-09-11T08:56:39.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2847906816 bytes 2024-09-11T08:56:39.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2895751168 bytes 2024-09-11T08:56:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2970045440 bytes 2024-09-11T08:56:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 3041986560 bytes 2024-09-11T08:56:40.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 3086785536 bytes 2024-09-11T08:56:40.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 3147268096 bytes 2024-09-11T08:56:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:56:40.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_34 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_34 snapshot 128M 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:40.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_34 128M 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:40.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_34 2024-09-11T08:56:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:56:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:56:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:41.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_34 snapshot 2024-09-11T08:56:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T08:56:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:41.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_34 snapshot 2024-09-11T08:56:41.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:56:41.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_34 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_34': 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 13fe2f03637e 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.13fe2f03637e 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:56:41.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:56:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:41 2024 2024-09-11T08:56:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:41 2024 2024-09-11T08:56:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:41 2024 2024-09-11T08:56:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:56:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:56:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 75f0a352-e3d0-46c7-8e9c-cd21648d227e 2024-09-11T08:56:41.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:56:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:56:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_34 2024-09-11T08:56:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:41.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T08:56:41.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=10 2024-09-11T08:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:56:41.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_34 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:56:42.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 84452352 bytes 2024-09-11T08:56:42.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 111617024 bytes 2024-09-11T08:56:42.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 166728704 bytes 2024-09-11T08:56:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 170086400 bytes 2024-09-11T08:56:42.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 177043456 bytes 2024-09-11T08:56:42.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 199883776 bytes 2024-09-11T08:56:43.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 220274688 bytes 2024-09-11T08:56:43.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 307207168 bytes 2024-09-11T08:56:43.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 369901568 bytes 2024-09-11T08:56:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 480156672 bytes 2024-09-11T08:56:44.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 517675008 bytes 2024-09-11T08:56:44.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 659587072 bytes 2024-09-11T08:56:44.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 676727808 bytes 2024-09-11T08:56:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 713608192 bytes 2024-09-11T08:56:44.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 765008896 bytes 2024-09-11T08:56:45.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 777464832 bytes 2024-09-11T08:56:45.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 804072448 bytes 2024-09-11T08:56:45.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 876600320 bytes 2024-09-11T08:56:45.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 884340736 bytes 2024-09-11T08:56:45.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 896878592 bytes 2024-09-11T08:56:45.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 916041728 bytes 2024-09-11T08:56:46.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 943464448 bytes 2024-09-11T08:56:46.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 977393664 bytes 2024-09-11T08:56:46.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1000613888 bytes 2024-09-11T08:56:46.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1037852672 bytes 2024-09-11T08:56:46.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1072912384 bytes 2024-09-11T08:56:46.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1096641536 bytes 2024-09-11T08:56:47.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1115854848 bytes 2024-09-11T08:56:47.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1134308352 bytes 2024-09-11T08:56:47.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1163232256 bytes 2024-09-11T08:56:47.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1221607424 bytes 2024-09-11T08:56:47.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1295096832 bytes 2024-09-11T08:56:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1334252544 bytes 2024-09-11T08:56:48.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1393481728 bytes 2024-09-11T08:56:48.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1459021824 bytes 2024-09-11T08:56:48.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1497873408 bytes 2024-09-11T08:56:48.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1522410496 bytes 2024-09-11T08:56:49.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1546317824 bytes 2024-09-11T08:56:49.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1610742784 bytes 2024-09-11T08:56:49.400 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1626784768 bytes 2024-09-11T08:56:49.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1637988352 bytes 2024-09-11T08:56:49.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1707476992 bytes 2024-09-11T08:56:49.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1773589504 bytes 2024-09-11T08:56:50.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1845348352 bytes 2024-09-11T08:56:50.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1856301056 bytes 2024-09-11T08:56:50.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1896683520 bytes 2024-09-11T08:56:50.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1920362496 bytes 2024-09-11T08:56:50.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1973047296 bytes 2024-09-11T08:56:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2034927616 bytes 2024-09-11T08:56:51.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2092416000 bytes 2024-09-11T08:56:51.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2108954624 bytes 2024-09-11T08:56:51.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2123747328 bytes 2024-09-11T08:56:51.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2182090752 bytes 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_35 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_35 snapshot 128M 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_35 128M 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T08:56:51.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:56:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:56:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:56:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:56:51.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_35 2024-09-11T08:56:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:56:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_35 snapshot 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:56:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_35 snapshot 2024-09-11T08:56:52.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:56:52.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_35 2024-09-11T08:56:52.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_35': 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14078978fc30 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14078978fc30 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:51 2024 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:51 2024 2024-09-11T08:56:52.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:51 2024 2024-09-11T08:56:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:56:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:56:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: d2627e7d-0b6b-467d-bdbf-be0af065f5f6 2024-09-11T08:56:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:56:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:56:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_35 2024-09-11T08:56:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:56:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:56:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T08:56:52.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:56:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:56:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:56:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_35 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:56:53.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 30925824 bytes 2024-09-11T08:56:53.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 78752768 bytes 2024-09-11T08:56:53.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 108974080 bytes 2024-09-11T08:56:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 136829952 bytes 2024-09-11T08:56:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 220459008 bytes 2024-09-11T08:56:54.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 297759744 bytes 2024-09-11T08:56:54.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 363084800 bytes 2024-09-11T08:56:54.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 381803520 bytes 2024-09-11T08:56:54.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 422426624 bytes 2024-09-11T08:56:54.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 506422272 bytes 2024-09-11T08:56:55.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 524819456 bytes 2024-09-11T08:56:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 596405248 bytes 2024-09-11T08:56:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 638918656 bytes 2024-09-11T08:56:55.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 683100160 bytes 2024-09-11T08:56:55.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 757807104 bytes 2024-09-11T08:56:56.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 804939776 bytes 2024-09-11T08:56:56.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 884069376 bytes 2024-09-11T08:56:56.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 914131968 bytes 2024-09-11T08:56:56.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 972102656 bytes 2024-09-11T08:56:57.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1061927936 bytes 2024-09-11T08:56:57.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1145960448 bytes 2024-09-11T08:56:57.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1225776128 bytes 2024-09-11T08:56:57.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1247898624 bytes 2024-09-11T08:56:57.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1290997760 bytes 2024-09-11T08:56:58.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1318809600 bytes 2024-09-11T08:56:58.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1344774144 bytes 2024-09-11T08:56:58.400 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1377601536 bytes 2024-09-11T08:56:58.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1411552256 bytes 2024-09-11T08:56:58.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1431518208 bytes 2024-09-11T08:56:58.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1463022592 bytes 2024-09-11T08:56:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1533059072 bytes 2024-09-11T08:56:59.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1562575872 bytes 2024-09-11T08:56:59.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1631424512 bytes 2024-09-11T08:56:59.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1665497088 bytes 2024-09-11T08:57:00.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1806173184 bytes 2024-09-11T08:57:00.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1806783488 bytes 2024-09-11T08:57:00.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1817450496 bytes 2024-09-11T08:57:00.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1922390016 bytes 2024-09-11T08:57:00.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1984071680 bytes 2024-09-11T08:57:00.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 2005918720 bytes 2024-09-11T08:57:01.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 2139612160 bytes 2024-09-11T08:57:01.493 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 2154173440 bytes 2024-09-11T08:57:01.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 2162248704 bytes 2024-09-11T08:57:01.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 2200401920 bytes 2024-09-11T08:57:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:57:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:57:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:57:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:57:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_36 2024-09-11T08:57:01.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_36 snapshot 128M 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_36 128M 2024-09-11T08:57:01.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:01.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_36 2024-09-11T08:57:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:57:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_36 snapshot 2024-09-11T08:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T08:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:01.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_36 snapshot 2024-09-11T08:57:02.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:57:02.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_36 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_36': 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 140f222a16ad 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.140f222a16ad 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:01 2024 2024-09-11T08:57:02.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:01 2024 2024-09-11T08:57:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:01 2024 2024-09-11T08:57:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:57:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:57:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: f6933439-7dab-4216-b4b5-b329ed22c0f6 2024-09-11T08:57:02.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:57:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:57:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_36 2024-09-11T08:57:02.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T08:57:02.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:57:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=14 2024-09-11T08:57:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:57:02.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 14s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_36 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:57:02.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 66545664 bytes 2024-09-11T08:57:02.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 122189824 bytes 2024-09-11T08:57:02.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 185007104 bytes 2024-09-11T08:57:03.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 246190080 bytes 2024-09-11T08:57:03.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 296327168 bytes 2024-09-11T08:57:03.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 381292544 bytes 2024-09-11T08:57:03.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 472749056 bytes 2024-09-11T08:57:04.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 530131968 bytes 2024-09-11T08:57:04.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 542400512 bytes 2024-09-11T08:57:04.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 543001600 bytes 2024-09-11T08:57:04.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 588487680 bytes 2024-09-11T08:57:04.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 636292096 bytes 2024-09-11T08:57:04.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 704370688 bytes 2024-09-11T08:57:05.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 731545600 bytes 2024-09-11T08:57:05.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 774387712 bytes 2024-09-11T08:57:05.509 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 830361600 bytes 2024-09-11T08:57:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 895042560 bytes 2024-09-11T08:57:05.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 907048960 bytes 2024-09-11T08:57:06.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1022516224 bytes 2024-09-11T08:57:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1048954880 bytes 2024-09-11T08:57:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1073891328 bytes 2024-09-11T08:57:06.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1128842240 bytes 2024-09-11T08:57:06.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1196257280 bytes 2024-09-11T08:57:07.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1245756416 bytes 2024-09-11T08:57:07.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1282039808 bytes 2024-09-11T08:57:07.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1322180608 bytes 2024-09-11T08:57:07.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1344046080 bytes 2024-09-11T08:57:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1399676928 bytes 2024-09-11T08:57:07.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1415513088 bytes 2024-09-11T08:57:08.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1492075520 bytes 2024-09-11T08:57:08.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1495950336 bytes 2024-09-11T08:57:08.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1551337472 bytes 2024-09-11T08:57:08.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1608581120 bytes 2024-09-11T08:57:08.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1661568000 bytes 2024-09-11T08:57:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1662182400 bytes 2024-09-11T08:57:09.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1709143040 bytes 2024-09-11T08:57:09.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1731490816 bytes 2024-09-11T08:57:09.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1777948672 bytes 2024-09-11T08:57:09.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1850383360 bytes 2024-09-11T08:57:10.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1887937536 bytes 2024-09-11T08:57:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1941146624 bytes 2024-09-11T08:57:10.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1962044416 bytes 2024-09-11T08:57:10.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 2001988608 bytes 2024-09-11T08:57:10.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 2028472320 bytes 2024-09-11T08:57:10.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2071876608 bytes 2024-09-11T08:57:11.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2112955392 bytes 2024-09-11T08:57:11.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2180899840 bytes 2024-09-11T08:57:11.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2240002048 bytes 2024-09-11T08:57:11.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2261169152 bytes 2024-09-11T08:57:11.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2309556224 bytes 2024-09-11T08:57:11.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2317103104 bytes 2024-09-11T08:57:12.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2333700096 bytes 2024-09-11T08:57:12.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2395563008 bytes 2024-09-11T08:57:12.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2462956544 bytes 2024-09-11T08:57:12.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2479637504 bytes 2024-09-11T08:57:12.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2518814720 bytes 2024-09-11T08:57:13.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2519439360 bytes 2024-09-11T08:57:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2559467520 bytes 2024-09-11T08:57:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2582201344 bytes 2024-09-11T08:57:13.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2585588736 bytes 2024-09-11T08:57:13.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2651935744 bytes 2024-09-11T08:57:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2705064960 bytes 2024-09-11T08:57:14.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2726362112 bytes 2024-09-11T08:57:14.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2769785856 bytes 2024-09-11T08:57:14.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2827104256 bytes 2024-09-11T08:57:14.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2875344896 bytes 2024-09-11T08:57:14.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2879918080 bytes 2024-09-11T08:57:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2924724224 bytes 2024-09-11T08:57:15.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2959792128 bytes 2024-09-11T08:57:15.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 3002728448 bytes 2024-09-11T08:57:15.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 3065262080 bytes 2024-09-11T08:57:15.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 3114943488 bytes 2024-09-11T08:57:15.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 3153467392 bytes 2024-09-11T08:57:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 3210387456 bytes 2024-09-11T08:57:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:57:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:57:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:57:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:57:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_37 2024-09-11T08:57:16.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_37 snapshot 128M 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_37 128M 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:16.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T08:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:16.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_37 2024-09-11T08:57:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:57:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:16.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_37 snapshot 2024-09-11T08:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T08:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:16.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_37 snapshot 2024-09-11T08:57:16.378 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:57:16.379 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:57:16.379 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T08:57:16.380 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:57:16.381+0000 7f98c6616640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 39789) UID: 1000 2024-09-11T08:57:16.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:57:16.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_37 2024-09-11T08:57:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_37': 2024-09-11T08:57:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:57:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:57:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:57:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14174972d8c4 2024-09-11T08:57:16.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14174972d8c4 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:16 2024 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:16 2024 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:16 2024 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a953381e-b730-4277-a246-b871eee2718e 2024-09-11T08:57:16.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:57:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:57:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_37 2024-09-11T08:57:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T08:57:16.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:57:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=11 2024-09-11T08:57:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:57:16.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_37 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:57:16.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 603136 bytes 2024-09-11T08:57:16.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 6317056 bytes 2024-09-11T08:57:17.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 56247296 bytes 2024-09-11T08:57:17.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 92493824 bytes 2024-09-11T08:57:17.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 142258176 bytes 2024-09-11T08:57:17.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 172922880 bytes 2024-09-11T08:57:17.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 216626176 bytes 2024-09-11T08:57:18.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 289647616 bytes 2024-09-11T08:57:18.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 395540480 bytes 2024-09-11T08:57:18.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 458591232 bytes 2024-09-11T08:57:18.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 501127168 bytes 2024-09-11T08:57:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 550845440 bytes 2024-09-11T08:57:19.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 627564544 bytes 2024-09-11T08:57:19.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 648641536 bytes 2024-09-11T08:57:19.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 667747328 bytes 2024-09-11T08:57:19.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 714140672 bytes 2024-09-11T08:57:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 783246336 bytes 2024-09-11T08:57:20.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 783877120 bytes 2024-09-11T08:57:20.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 804427776 bytes 2024-09-11T08:57:20.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 840581120 bytes 2024-09-11T08:57:20.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 908024832 bytes 2024-09-11T08:57:20.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 915270656 bytes 2024-09-11T08:57:21.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 991054848 bytes 2024-09-11T08:57:21.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1050697728 bytes 2024-09-11T08:57:21.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1051330560 bytes 2024-09-11T08:57:21.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1121913856 bytes 2024-09-11T08:57:21.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1126749184 bytes 2024-09-11T08:57:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1246792704 bytes 2024-09-11T08:57:22.259 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1285427200 bytes 2024-09-11T08:57:22.380 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:57:22.380 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:57:22.381 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:57:22.381 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:57:22.381 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:57:22.383 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:57:22.384 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-09-11T08:57:22.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1317043200 bytes 2024-09-11T08:57:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1334150144 bytes 2024-09-11T08:57:22.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1375403008 bytes 2024-09-11T08:57:23.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1473839104 bytes 2024-09-11T08:57:23.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1514238976 bytes 2024-09-11T08:57:23.487 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1588400128 bytes 2024-09-11T08:57:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1678918656 bytes 2024-09-11T08:57:23.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1679545344 bytes 2024-09-11T08:57:24.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1717558272 bytes 2024-09-11T08:57:24.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1778062336 bytes 2024-09-11T08:57:24.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1857494016 bytes 2024-09-11T08:57:24.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1881841664 bytes 2024-09-11T08:57:24.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1916477440 bytes 2024-09-11T08:57:24.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1945848832 bytes 2024-09-11T08:57:25.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1947891712 bytes 2024-09-11T08:57:25.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2023225344 bytes 2024-09-11T08:57:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2051691520 bytes 2024-09-11T08:57:25.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2096808960 bytes 2024-09-11T08:57:25.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2172462080 bytes 2024-09-11T08:57:26.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2200858624 bytes 2024-09-11T08:57:26.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2245643264 bytes 2024-09-11T08:57:26.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2281394176 bytes 2024-09-11T08:57:26.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2344881152 bytes 2024-09-11T08:57:26.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2404999168 bytes 2024-09-11T08:57:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2467464192 bytes 2024-09-11T08:57:27.270 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4486013100799945) > weight (0.25) 2024-09-11T08:57:27.271 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.3528865511032411) > weight (0.25) 2024-09-11T08:57:27.271 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.2 2024-09-11T08:57:27.271 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Sent signal 15 2024-09-11T08:57:27.272 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T08:57:27.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T08:57:27.273+0000 7fd66c03c640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 31649) UID: 1000 2024-09-11T08:57:27.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2536984576 bytes 2024-09-11T08:57:27.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2578800640 bytes 2024-09-11T08:57:27.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_38 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_38 snapshot 128M 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_38 128M 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T08:57:27.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:57:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:57:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:57:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:27.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_38 2024-09-11T08:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:27.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_38 snapshot 2024-09-11T08:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T08:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:27.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_38 snapshot 2024-09-11T08:57:28.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:57:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_38 2024-09-11T08:57:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_38': 2024-09-11T08:57:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:57:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:57:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:57:28.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1427afcfda82 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1427afcfda82 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:27 2024 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:27 2024 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:27 2024 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: e269182c-689f-4c97-bd46-05795a725cb4 2024-09-11T08:57:28.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:57:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:57:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_38 2024-09-11T08:57:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:28.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T08:57:28.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:57:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:57:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:57:28.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_38 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:57:28.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 20441088 bytes 2024-09-11T08:57:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 29262848 bytes 2024-09-11T08:57:29.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 64498688 bytes 2024-09-11T08:57:29.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 108796928 bytes 2024-09-11T08:57:29.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 148147200 bytes 2024-09-11T08:57:29.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 178752512 bytes 2024-09-11T08:57:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 220403712 bytes 2024-09-11T08:57:29.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 228924416 bytes 2024-09-11T08:57:30.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 269582336 bytes 2024-09-11T08:57:30.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 293894144 bytes 2024-09-11T08:57:30.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 311599104 bytes 2024-09-11T08:57:30.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 314260480 bytes 2024-09-11T08:57:30.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 427310080 bytes 2024-09-11T08:57:31.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 473407488 bytes 2024-09-11T08:57:31.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 511160320 bytes 2024-09-11T08:57:31.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 594140160 bytes 2024-09-11T08:57:31.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 628550656 bytes 2024-09-11T08:57:31.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 654331904 bytes 2024-09-11T08:57:32.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 694005760 bytes 2024-09-11T08:57:32.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 742651904 bytes 2024-09-11T08:57:32.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 796427264 bytes 2024-09-11T08:57:32.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 871910400 bytes 2024-09-11T08:57:32.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 917807104 bytes 2024-09-11T08:57:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 944080896 bytes 2024-09-11T08:57:33.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 945258496 bytes 2024-09-11T08:57:33.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 989881344 bytes 2024-09-11T08:57:33.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1000521728 bytes 2024-09-11T08:57:33.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1016758272 bytes 2024-09-11T08:57:33.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1056872448 bytes 2024-09-11T08:57:33.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1086777344 bytes 2024-09-11T08:57:34.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1145255936 bytes 2024-09-11T08:57:34.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1196950528 bytes 2024-09-11T08:57:34.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1210756096 bytes 2024-09-11T08:57:34.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1266704384 bytes 2024-09-11T08:57:34.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1306544128 bytes 2024-09-11T08:57:35.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1347183616 bytes 2024-09-11T08:57:35.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1347788800 bytes 2024-09-11T08:57:35.273 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.2 2024-09-11T08:57:35.273 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:57:35.273 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.2 2024-09-11T08:57:35.273 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Restarting daemon 2024-09-11T08:57:35.274 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 2024-09-11T08:57:35.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Started 2024-09-11T08:57:35.276 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-11T08:57:35.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1391060992 bytes 2024-09-11T08:57:35.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1451650048 bytes 2024-09-11T08:57:35.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_39 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_39 snapshot 128M 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_39 128M 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:57:35.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:57:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:57:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:35.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_39 2024-09-11T08:57:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:57:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:57:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:35.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_39 snapshot 2024-09-11T08:57:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T08:57:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:35.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_39 snapshot 2024-09-11T08:57:36.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:57:36.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_39 2024-09-11T08:57:36.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_39': 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 143c57bfe5fa 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.143c57bfe5fa 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:35 2024 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:35 2024 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:35 2024 2024-09-11T08:57:36.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:57:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:57:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: c046011c-40fa-4138-bba6-2fc652d532ec 2024-09-11T08:57:36.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:57:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:57:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_39 2024-09-11T08:57:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:36.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T08:57:36.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:57:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=8 2024-09-11T08:57:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:57:36.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_39 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:57:36.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 26302464 bytes 2024-09-11T08:57:37.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 78659584 bytes 2024-09-11T08:57:37.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 156981248 bytes 2024-09-11T08:57:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 176861184 bytes 2024-09-11T08:57:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 213897216 bytes 2024-09-11T08:57:37.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 259249152 bytes 2024-09-11T08:57:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 297931776 bytes 2024-09-11T08:57:38.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 449832960 bytes 2024-09-11T08:57:38.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 525996032 bytes 2024-09-11T08:57:38.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 573117440 bytes 2024-09-11T08:57:39.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 596720640 bytes 2024-09-11T08:57:39.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 667023360 bytes 2024-09-11T08:57:39.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 750180352 bytes 2024-09-11T08:57:39.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 870894592 bytes 2024-09-11T08:57:39.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 904941568 bytes 2024-09-11T08:57:40.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 943907840 bytes 2024-09-11T08:57:40.365 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 1002762240 bytes 2024-09-11T08:57:40.493 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 1019148288 bytes 2024-09-11T08:57:40.596 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1020388352 bytes 2024-09-11T08:57:40.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1099943936 bytes 2024-09-11T08:57:40.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1100559360 bytes 2024-09-11T08:57:41.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1242943488 bytes 2024-09-11T08:57:41.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1284213760 bytes 2024-09-11T08:57:41.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1343495168 bytes 2024-09-11T08:57:41.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1361121280 bytes 2024-09-11T08:57:42.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1442513920 bytes 2024-09-11T08:57:42.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1477864448 bytes 2024-09-11T08:57:42.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1481933824 bytes 2024-09-11T08:57:42.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1545459712 bytes 2024-09-11T08:57:42.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1562339328 bytes 2024-09-11T08:57:42.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1587529728 bytes 2024-09-11T08:57:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1588124672 bytes 2024-09-11T08:57:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1592830976 bytes 2024-09-11T08:57:43.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1614022656 bytes 2024-09-11T08:57:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1653331968 bytes 2024-09-11T08:57:43.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1716067328 bytes 2024-09-11T08:57:43.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1739750400 bytes 2024-09-11T08:57:43.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1765271552 bytes 2024-09-11T08:57:44.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1819738112 bytes 2024-09-11T08:57:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1865702400 bytes 2024-09-11T08:57:44.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1948369920 bytes 2024-09-11T08:57:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:57:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:57:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:57:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:57:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_40 2024-09-11T08:57:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_40 snapshot 128M 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_40 128M 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T08:57:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:57:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:57:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:57:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:44.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_40 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_40 snapshot 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T08:57:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_40 snapshot 2024-09-11T08:57:45.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:57:45.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_40 2024-09-11T08:57:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_40': 2024-09-11T08:57:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:57:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:57:45.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 144822a6d91a 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.144822a6d91a 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:44 2024 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:44 2024 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:44 2024 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 24872562-d2cb-4569-88af-e8992d096531 2024-09-11T08:57:45.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:57:45.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:57:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_40 2024-09-11T08:57:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T08:57:45.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:57:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=12 2024-09-11T08:57:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:57:45.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_40 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:57:46.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 16786432 bytes 2024-09-11T08:57:46.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 30264320 bytes 2024-09-11T08:57:46.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 45989888 bytes 2024-09-11T08:57:46.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 137244672 bytes 2024-09-11T08:57:46.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 190032896 bytes 2024-09-11T08:57:47.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 209233920 bytes 2024-09-11T08:57:47.333 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 277903360 bytes 2024-09-11T08:57:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 295552000 bytes 2024-09-11T08:57:47.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 308011008 bytes 2024-09-11T08:57:47.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 344905728 bytes 2024-09-11T08:57:47.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 381105152 bytes 2024-09-11T08:57:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 471207936 bytes 2024-09-11T08:57:48.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 492920832 bytes 2024-09-11T08:57:48.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 572636160 bytes 2024-09-11T08:57:48.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 573263872 bytes 2024-09-11T08:57:48.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 614499328 bytes 2024-09-11T08:57:49.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 680735744 bytes 2024-09-11T08:57:49.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 682750976 bytes 2024-09-11T08:57:49.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 741513216 bytes 2024-09-11T08:57:49.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 779649024 bytes 2024-09-11T08:57:49.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 797768704 bytes 2024-09-11T08:57:50.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 945951744 bytes 2024-09-11T08:57:50.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 961977344 bytes 2024-09-11T08:57:50.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1027122176 bytes 2024-09-11T08:57:50.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1079011328 bytes 2024-09-11T08:57:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1106188288 bytes 2024-09-11T08:57:51.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1110961152 bytes 2024-09-11T08:57:51.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1161528320 bytes 2024-09-11T08:57:51.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1233654784 bytes 2024-09-11T08:57:51.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1244433408 bytes 2024-09-11T08:57:51.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1270447104 bytes 2024-09-11T08:57:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1320364032 bytes 2024-09-11T08:57:52.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1389554688 bytes 2024-09-11T08:57:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1404521472 bytes 2024-09-11T08:57:52.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1498714112 bytes 2024-09-11T08:57:52.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1543244800 bytes 2024-09-11T08:57:52.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1548241920 bytes 2024-09-11T08:57:53.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1548836864 bytes 2024-09-11T08:57:53.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1596683264 bytes 2024-09-11T08:57:53.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1622657024 bytes 2024-09-11T08:57:53.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1638786048 bytes 2024-09-11T08:57:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1729746944 bytes 2024-09-11T08:57:53.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1782559744 bytes 2024-09-11T08:57:54.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1805473792 bytes 2024-09-11T08:57:54.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1840154624 bytes 2024-09-11T08:57:54.525 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1918984192 bytes 2024-09-11T08:57:54.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1964057600 bytes 2024-09-11T08:57:54.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1985701888 bytes 2024-09-11T08:57:55.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1995319296 bytes 2024-09-11T08:57:55.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2051965952 bytes 2024-09-11T08:57:55.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2154098688 bytes 2024-09-11T08:57:55.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2166321152 bytes 2024-09-11T08:57:55.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2224693248 bytes 2024-09-11T08:57:56.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2270916608 bytes 2024-09-11T08:57:56.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2271696896 bytes 2024-09-11T08:57:56.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2308342784 bytes 2024-09-11T08:57:56.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2335174656 bytes 2024-09-11T08:57:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2373207040 bytes 2024-09-11T08:57:56.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2413246464 bytes 2024-09-11T08:57:57.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2460112896 bytes 2024-09-11T08:57:57.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2520112128 bytes 2024-09-11T08:57:57.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2535922688 bytes 2024-09-11T08:57:57.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2576777216 bytes 2024-09-11T08:57:57.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2666903552 bytes 2024-09-11T08:57:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:57:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:57:57.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_41 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_41 snapshot 128M 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:57.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_41 128M 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:57:57.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_41 2024-09-11T08:57:58.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_41 snapshot 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:57:58.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_41 snapshot 2024-09-11T08:57:58.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:57:58.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_41 2024-09-11T08:57:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_41': 2024-09-11T08:57:58.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1454506271e4 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1454506271e4 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:57 2024 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:57 2024 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:57 2024 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:57:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:57:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: e18f305c-3600-4a05-ac01-cbd0ec5179dd 2024-09-11T08:57:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:57:58.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:57:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_41 2024-09-11T08:57:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:57:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:57:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T08:57:58.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:57:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=11 2024-09-11T08:57:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:57:58.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_41 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:57:59.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 36553728 bytes 2024-09-11T08:57:59.359 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 161796096 bytes 2024-09-11T08:57:59.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 172680192 bytes 2024-09-11T08:57:59.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 205020160 bytes 2024-09-11T08:57:59.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 231096320 bytes 2024-09-11T08:58:00.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 272363520 bytes 2024-09-11T08:58:00.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 288939008 bytes 2024-09-11T08:58:00.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 341816320 bytes 2024-09-11T08:58:00.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 367009792 bytes 2024-09-11T08:58:00.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 453351424 bytes 2024-09-11T08:58:01.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 493853696 bytes 2024-09-11T08:58:01.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 539856896 bytes 2024-09-11T08:58:01.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 581549056 bytes 2024-09-11T08:58:01.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 591368192 bytes 2024-09-11T08:58:01.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 629736448 bytes 2024-09-11T08:58:01.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 663446528 bytes 2024-09-11T08:58:01.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 696623104 bytes 2024-09-11T08:58:02.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 743415808 bytes 2024-09-11T08:58:02.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 758433792 bytes 2024-09-11T08:58:02.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 779045888 bytes 2024-09-11T08:58:02.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 836169728 bytes 2024-09-11T08:58:02.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 859478016 bytes 2024-09-11T08:58:03.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 920424448 bytes 2024-09-11T08:58:03.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 948714496 bytes 2024-09-11T08:58:03.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 962345984 bytes 2024-09-11T08:58:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 964926464 bytes 2024-09-11T08:58:03.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1054743552 bytes 2024-09-11T08:58:03.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1091956736 bytes 2024-09-11T08:58:04.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1120535552 bytes 2024-09-11T08:58:04.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1161838592 bytes 2024-09-11T08:58:04.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1244612608 bytes 2024-09-11T08:58:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1321930752 bytes 2024-09-11T08:58:04.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1375202304 bytes 2024-09-11T08:58:05.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1406636032 bytes 2024-09-11T08:58:05.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1441062912 bytes 2024-09-11T08:58:05.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1484221440 bytes 2024-09-11T08:58:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1541671936 bytes 2024-09-11T08:58:05.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1590584320 bytes 2024-09-11T08:58:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1622431744 bytes 2024-09-11T08:58:06.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1639694336 bytes 2024-09-11T08:58:06.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1781536768 bytes 2024-09-11T08:58:06.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1822994432 bytes 2024-09-11T08:58:06.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1876292608 bytes 2024-09-11T08:58:07.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1955651584 bytes 2024-09-11T08:58:07.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2013511680 bytes 2024-09-11T08:58:07.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2037197824 bytes 2024-09-11T08:58:07.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2071936000 bytes 2024-09-11T08:58:07.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2087468032 bytes 2024-09-11T08:58:08.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2098594816 bytes 2024-09-11T08:58:08.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2146138112 bytes 2024-09-11T08:58:08.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2172933120 bytes 2024-09-11T08:58:08.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2214082560 bytes 2024-09-11T08:58:08.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2260637696 bytes 2024-09-11T08:58:09.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2330500096 bytes 2024-09-11T08:58:09.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2346902528 bytes 2024-09-11T08:58:09.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2375929856 bytes 2024-09-11T08:58:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2414658560 bytes 2024-09-11T08:58:09.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2438988800 bytes 2024-09-11T08:58:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:58:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:58:09.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_42 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_42 snapshot 128M 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:09.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_42 128M 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:09.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_42 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_42 snapshot 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T08:58:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:09.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_42 snapshot 2024-09-11T08:58:10.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:58:10.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_42 2024-09-11T08:58:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_42': 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14609706c9fc 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14609706c9fc 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:09 2024 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:09 2024 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:09 2024 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:58:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:58:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 52286b49-eaf5-465a-988f-e2ac13d2e71c 2024-09-11T08:58:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:58:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:58:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_42 2024-09-11T08:58:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T08:58:10.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:58:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T08:58:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:58:10.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_42 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:58:11.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 37014528 bytes 2024-09-11T08:58:11.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 77299712 bytes 2024-09-11T08:58:11.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 129802240 bytes 2024-09-11T08:58:11.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 155298816 bytes 2024-09-11T08:58:11.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 206949376 bytes 2024-09-11T08:58:12.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 256933888 bytes 2024-09-11T08:58:12.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 286437376 bytes 2024-09-11T08:58:12.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 326908928 bytes 2024-09-11T08:58:12.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 385169408 bytes 2024-09-11T08:58:12.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 405819392 bytes 2024-09-11T08:58:12.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 479358976 bytes 2024-09-11T08:58:13.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 560425984 bytes 2024-09-11T08:58:13.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 626214912 bytes 2024-09-11T08:58:13.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 627679232 bytes 2024-09-11T08:58:13.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 646139904 bytes 2024-09-11T08:58:13.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 679699456 bytes 2024-09-11T08:58:14.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 733378560 bytes 2024-09-11T08:58:14.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 839611392 bytes 2024-09-11T08:58:14.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 871120896 bytes 2024-09-11T08:58:14.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 914095104 bytes 2024-09-11T08:58:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 942976000 bytes 2024-09-11T08:58:15.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 966515712 bytes 2024-09-11T08:58:15.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1004662784 bytes 2024-09-11T08:58:15.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1065018368 bytes 2024-09-11T08:58:15.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1139076096 bytes 2024-09-11T08:58:15.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1179127808 bytes 2024-09-11T08:58:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:58:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:58:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:58:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:58:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_43 2024-09-11T08:58:15.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_43 snapshot 128M 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_43 128M 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:15.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:15.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_43 2024-09-11T08:58:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:58:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:15.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:58:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_43 snapshot 2024-09-11T08:58:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T08:58:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:15.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_43 snapshot 2024-09-11T08:58:16.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:58:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_43 2024-09-11T08:58:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_43': 2024-09-11T08:58:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:58:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:58:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:58:16.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 146c533d3301 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.146c533d3301 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:15 2024 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:15 2024 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:15 2024 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 540fcbee-d8ed-438b-8b66-b3b65770db9f 2024-09-11T08:58:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:58:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:58:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_43 2024-09-11T08:58:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T08:58:16.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:58:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T08:58:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:58:16.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_43 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:58:17.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 54540288 bytes 2024-09-11T08:58:17.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 135811072 bytes 2024-09-11T08:58:17.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 240256000 bytes 2024-09-11T08:58:18.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 309965824 bytes 2024-09-11T08:58:18.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 337011712 bytes 2024-09-11T08:58:18.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 408491008 bytes 2024-09-11T08:58:18.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 462535680 bytes 2024-09-11T08:58:18.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 514080768 bytes 2024-09-11T08:58:19.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 542375936 bytes 2024-09-11T08:58:19.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 576831488 bytes 2024-09-11T08:58:19.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 585421824 bytes 2024-09-11T08:58:19.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 635751424 bytes 2024-09-11T08:58:19.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 698684416 bytes 2024-09-11T08:58:20.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 748512256 bytes 2024-09-11T08:58:20.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 763039744 bytes 2024-09-11T08:58:20.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 804753408 bytes 2024-09-11T08:58:20.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 877486080 bytes 2024-09-11T08:58:20.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 929608704 bytes 2024-09-11T08:58:20.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 956346368 bytes 2024-09-11T08:58:21.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1045951488 bytes 2024-09-11T08:58:21.375 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1046584320 bytes 2024-09-11T08:58:21.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1089468416 bytes 2024-09-11T08:58:21.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1110297600 bytes 2024-09-11T08:58:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1178861568 bytes 2024-09-11T08:58:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:58:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:58:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:58:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_44 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_44 snapshot 128M 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_44 128M 2024-09-11T08:58:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:21.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_44 2024-09-11T08:58:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_44 snapshot 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:22.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_44 snapshot 2024-09-11T08:58:22.525 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:58:22.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_44 2024-09-11T08:58:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_44': 2024-09-11T08:58:22.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1478bdba86ad 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1478bdba86ad 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:22 2024 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:22 2024 2024-09-11T08:58:22.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:22 2024 2024-09-11T08:58:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:58:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:58:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 9eba4b2e-6f4b-4273-9365-f3717576509c 2024-09-11T08:58:22.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:58:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:58:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_44 2024-09-11T08:58:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:22.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T08:58:22.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:58:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T08:58:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:58:22.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_44 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:58:22.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 38546432 bytes 2024-09-11T08:58:23.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 90455040 bytes 2024-09-11T08:58:23.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 121850880 bytes 2024-09-11T08:58:23.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 164297728 bytes 2024-09-11T08:58:23.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 224869376 bytes 2024-09-11T08:58:23.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 287350784 bytes 2024-09-11T08:58:24.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 414427136 bytes 2024-09-11T08:58:24.358 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 477472768 bytes 2024-09-11T08:58:24.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 567370752 bytes 2024-09-11T08:58:24.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 619181056 bytes 2024-09-11T08:58:24.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 626891776 bytes 2024-09-11T08:58:25.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 663055360 bytes 2024-09-11T08:58:25.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 703541248 bytes 2024-09-11T08:58:25.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 716939264 bytes 2024-09-11T08:58:25.519 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 729715712 bytes 2024-09-11T08:58:25.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 756120576 bytes 2024-09-11T08:58:25.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 793265152 bytes 2024-09-11T08:58:26.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 889701376 bytes 2024-09-11T08:58:26.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 947902464 bytes 2024-09-11T08:58:26.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 980218880 bytes 2024-09-11T08:58:26.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 997600256 bytes 2024-09-11T08:58:26.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1083296768 bytes 2024-09-11T08:58:27.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1152098304 bytes 2024-09-11T08:58:27.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1200637952 bytes 2024-09-11T08:58:27.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1241040896 bytes 2024-09-11T08:58:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1280666624 bytes 2024-09-11T08:58:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1324322816 bytes 2024-09-11T08:58:28.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1380792320 bytes 2024-09-11T08:58:28.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1432469504 bytes 2024-09-11T08:58:28.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1481911296 bytes 2024-09-11T08:58:28.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1526377472 bytes 2024-09-11T08:58:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1547116544 bytes 2024-09-11T08:58:28.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1597799424 bytes 2024-09-11T08:58:29.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1625362432 bytes 2024-09-11T08:58:29.364 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1686299648 bytes 2024-09-11T08:58:29.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1716744192 bytes 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_45 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_45 snapshot 128M 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_45 128M 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:58:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:58:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:58:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:29.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_45 2024-09-11T08:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_45 snapshot 2024-09-11T08:58:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T08:58:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:29.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_45 snapshot 2024-09-11T08:58:30.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:58:30.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_45 2024-09-11T08:58:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_45': 2024-09-11T08:58:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:58:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:58:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:58:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1484c165f271 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1484c165f271 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:29 2024 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:29 2024 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:29 2024 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: f5150c59-0ae1-4502-a239-7a0369bc62f5 2024-09-11T08:58:30.610 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:58:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:58:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_45 2024-09-11T08:58:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:30.613 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T08:58:30.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:58:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:58:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:58:30.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_45 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:58:30.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 29105152 bytes 2024-09-11T08:58:30.835 INFO:tasks.ceph.cluster2.osd.0.smithi077.stderr:2024-09-11T08:58:30.835+0000 7f2b03c68640 -1 reset not still connected to 0x55c649d9a750 2024-09-11T08:58:30.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 72509440 bytes 2024-09-11T08:58:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 149211136 bytes 2024-09-11T08:58:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 195090432 bytes 2024-09-11T08:58:31.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 285362176 bytes 2024-09-11T08:58:31.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 326856704 bytes 2024-09-11T08:58:32.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 439211008 bytes 2024-09-11T08:58:32.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 507235328 bytes 2024-09-11T08:58:32.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 534714368 bytes 2024-09-11T08:58:32.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 563204096 bytes 2024-09-11T08:58:32.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 601661440 bytes 2024-09-11T08:58:33.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 689503232 bytes 2024-09-11T08:58:33.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 690108416 bytes 2024-09-11T08:58:33.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 698138624 bytes 2024-09-11T08:58:33.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 730847232 bytes 2024-09-11T08:58:33.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 736396288 bytes 2024-09-11T08:58:33.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 760868864 bytes 2024-09-11T08:58:34.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 820746240 bytes 2024-09-11T08:58:34.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 883329024 bytes 2024-09-11T08:58:34.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 903711744 bytes 2024-09-11T08:58:34.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 942203904 bytes 2024-09-11T08:58:34.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 958512128 bytes 2024-09-11T08:58:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1041030144 bytes 2024-09-11T08:58:35.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1084041216 bytes 2024-09-11T08:58:35.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1118804992 bytes 2024-09-11T08:58:35.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1119413248 bytes 2024-09-11T08:58:35.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1126875136 bytes 2024-09-11T08:58:35.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1139371008 bytes 2024-09-11T08:58:35.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1192967168 bytes 2024-09-11T08:58:36.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1263104000 bytes 2024-09-11T08:58:36.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1282674688 bytes 2024-09-11T08:58:36.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1286915072 bytes 2024-09-11T08:58:36.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1322056704 bytes 2024-09-11T08:58:36.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1388724224 bytes 2024-09-11T08:58:36.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1393500160 bytes 2024-09-11T08:58:37.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1407498240 bytes 2024-09-11T08:58:37.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1455760384 bytes 2024-09-11T08:58:37.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1505655808 bytes 2024-09-11T08:58:37.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1522700288 bytes 2024-09-11T08:58:37.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1566300160 bytes 2024-09-11T08:58:37.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1616235520 bytes 2024-09-11T08:58:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1631244288 bytes 2024-09-11T08:58:38.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1657758720 bytes 2024-09-11T08:58:38.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1688652800 bytes 2024-09-11T08:58:38.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1716072448 bytes 2024-09-11T08:58:38.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1729509376 bytes 2024-09-11T08:58:38.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1763714048 bytes 2024-09-11T08:58:39.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1829632000 bytes 2024-09-11T08:58:39.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1877720064 bytes 2024-09-11T08:58:39.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1878920192 bytes 2024-09-11T08:58:39.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 1930461184 bytes 2024-09-11T08:58:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 1939282944 bytes 2024-09-11T08:58:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:58:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:58:39.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_46 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_46 snapshot 128M 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_46 128M 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:39.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T08:58:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:58:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:58:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:58:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_46 2024-09-11T08:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:39.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_46 snapshot 2024-09-11T08:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T08:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:39.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_46 snapshot 2024-09-11T08:58:40.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:58:40.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_46 2024-09-11T08:58:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_46': 2024-09-11T08:58:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:58:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:58:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:58:40.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14908a4a8ebe 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14908a4a8ebe 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:39 2024 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:39 2024 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:39 2024 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8cd87261-11aa-45a3-8d2c-211e06dbeb62 2024-09-11T08:58:40.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:58:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:58:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_46 2024-09-11T08:58:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:40.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T08:58:40.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:58:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T08:58:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:58:40.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_46 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:58:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 34974720 bytes 2024-09-11T08:58:41.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 102363136 bytes 2024-09-11T08:58:41.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 219176960 bytes 2024-09-11T08:58:41.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 234491904 bytes 2024-09-11T08:58:41.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 264033280 bytes 2024-09-11T08:58:41.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 266153984 bytes 2024-09-11T08:58:41.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 292234240 bytes 2024-09-11T08:58:42.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 300193792 bytes 2024-09-11T08:58:42.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 342488064 bytes 2024-09-11T08:58:42.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 379568128 bytes 2024-09-11T08:58:42.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 392328192 bytes 2024-09-11T08:58:42.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 431048704 bytes 2024-09-11T08:58:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 466152448 bytes 2024-09-11T08:58:43.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 532393984 bytes 2024-09-11T08:58:43.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 553935872 bytes 2024-09-11T08:58:43.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 606071808 bytes 2024-09-11T08:58:43.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 655466496 bytes 2024-09-11T08:58:43.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 659089408 bytes 2024-09-11T08:58:43.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 682435584 bytes 2024-09-11T08:58:44.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 768665600 bytes 2024-09-11T08:58:44.267 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 789485568 bytes 2024-09-11T08:58:44.277 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9515892558452659) > weight (0.25) 2024-09-11T08:58:44.278 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T08:58:44.278 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T08:58:44.278 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-11T08:58:44.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T08:58:44.279+0000 7ff586681640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 42131) UID: 1000 2024-09-11T08:58:44.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 828930048 bytes 2024-09-11T08:58:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 841238528 bytes 2024-09-11T08:58:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 855499776 bytes 2024-09-11T08:58:44.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 929408000 bytes 2024-09-11T08:58:45.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 941027328 bytes 2024-09-11T08:58:45.279 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T08:58:45.279 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:58:45.279 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T08:58:45.279 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T08:58:45.280 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T08:58:45.282 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T08:58:45.282 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-11T08:58:45.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1008440320 bytes 2024-09-11T08:58:45.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1031585792 bytes 2024-09-11T08:58:45.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1069104128 bytes 2024-09-11T08:58:45.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1088956416 bytes 2024-09-11T08:58:45.841 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:58:45.843+0000 7f00d11de640 -1 reset not still connected to 0x55b6187db6c0 2024-09-11T08:58:45.843 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T08:58:45.843+0000 7f00d11de640 -1 reset not still connected to 0x55b6282fc9c0 2024-09-11T08:58:45.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1138711552 bytes 2024-09-11T08:58:46.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1202603008 bytes 2024-09-11T08:58:46.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1239513088 bytes 2024-09-11T08:58:46.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1271180288 bytes 2024-09-11T08:58:46.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1297082368 bytes 2024-09-11T08:58:46.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1364144128 bytes 2024-09-11T08:58:47.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1376917504 bytes 2024-09-11T08:58:47.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1377541120 bytes 2024-09-11T08:58:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1408780288 bytes 2024-09-11T08:58:47.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1423297536 bytes 2024-09-11T08:58:47.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1472708608 bytes 2024-09-11T08:58:47.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1514574848 bytes 2024-09-11T08:58:48.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1600285696 bytes 2024-09-11T08:58:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1609961472 bytes 2024-09-11T08:58:48.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1642989568 bytes 2024-09-11T08:58:48.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1701850112 bytes 2024-09-11T08:58:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1719463936 bytes 2024-09-11T08:58:49.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1747072000 bytes 2024-09-11T08:58:49.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1784503296 bytes 2024-09-11T08:58:49.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1830290432 bytes 2024-09-11T08:58:49.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 1879007232 bytes 2024-09-11T08:58:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:58:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:58:49.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_47 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_47 snapshot 128M 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_47 128M 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:58:49.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_47 2024-09-11T08:58:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:58:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:58:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:58:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_47 snapshot 2024-09-11T08:58:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:49.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T08:58:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:58:49.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_47 snapshot 2024-09-11T08:58:50.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:58:50.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_47 2024-09-11T08:58:50.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_47': 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14a5d7388409 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14a5d7388409 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:49 2024 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:49 2024 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:49 2024 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:58:50.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:58:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8ad010f4-22cf-4eef-8e81-0a6236a13192 2024-09-11T08:58:50.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:58:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:58:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_47 2024-09-11T08:58:50.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T08:58:50.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:58:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=10 2024-09-11T08:58:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:58:50.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_47 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:58:50.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 24479744 bytes 2024-09-11T08:58:51.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 78898176 bytes 2024-09-11T08:58:51.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 119300096 bytes 2024-09-11T08:58:51.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 143635456 bytes 2024-09-11T08:58:51.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 163437568 bytes 2024-09-11T08:58:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 240088064 bytes 2024-09-11T08:58:51.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 265079808 bytes 2024-09-11T08:58:52.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 351498240 bytes 2024-09-11T08:58:52.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 457833472 bytes 2024-09-11T08:58:52.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 479696896 bytes 2024-09-11T08:58:52.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 543248384 bytes 2024-09-11T08:58:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 585308160 bytes 2024-09-11T08:58:53.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 645154816 bytes 2024-09-11T08:58:53.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 700386304 bytes 2024-09-11T08:58:53.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 701015040 bytes 2024-09-11T08:58:53.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 741173248 bytes 2024-09-11T08:58:54.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 844455936 bytes 2024-09-11T08:58:54.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 855044096 bytes 2024-09-11T08:58:54.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 888530944 bytes 2024-09-11T08:58:54.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 889132032 bytes 2024-09-11T08:58:54.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 996748288 bytes 2024-09-11T08:58:55.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1069510656 bytes 2024-09-11T08:58:55.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1119944704 bytes 2024-09-11T08:58:55.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1162662912 bytes 2024-09-11T08:58:55.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1223187456 bytes 2024-09-11T08:58:55.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1244762112 bytes 2024-09-11T08:58:56.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1350720512 bytes 2024-09-11T08:58:56.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1419074560 bytes 2024-09-11T08:58:56.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1484217344 bytes 2024-09-11T08:58:56.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1500123136 bytes 2024-09-11T08:58:56.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1552101376 bytes 2024-09-11T08:58:57.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1599181824 bytes 2024-09-11T08:58:57.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1626507264 bytes 2024-09-11T08:58:57.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1650547712 bytes 2024-09-11T08:58:57.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1655091200 bytes 2024-09-11T08:58:57.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1753720832 bytes 2024-09-11T08:58:57.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1762714624 bytes 2024-09-11T08:58:57.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1779979264 bytes 2024-09-11T08:58:58.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1808230400 bytes 2024-09-11T08:58:58.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1816678400 bytes 2024-09-11T08:58:58.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1832265728 bytes 2024-09-11T08:58:58.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1847177216 bytes 2024-09-11T08:58:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1871566848 bytes 2024-09-11T08:58:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1897929728 bytes 2024-09-11T08:58:58.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1945955328 bytes 2024-09-11T08:58:59.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1986162688 bytes 2024-09-11T08:58:59.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2006704128 bytes 2024-09-11T08:58:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2077156352 bytes 2024-09-11T08:58:59.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2077760512 bytes 2024-09-11T08:58:59.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2120358912 bytes 2024-09-11T08:59:00.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2195410944 bytes 2024-09-11T08:59:00.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2245056512 bytes 2024-09-11T08:59:00.384 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T08:59:00.385 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T08:59:00.385 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T08:59:00.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T08:59:00.387+0000 7fc075b64640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 42647) UID: 1000 2024-09-11T08:59:00.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2265860096 bytes 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_48 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_48 snapshot 128M 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:00.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_48 128M 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:59:00.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:59:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:59:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:00.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_48 2024-09-11T08:59:00.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:59:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:59:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:00.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_48 snapshot 2024-09-11T08:59:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T08:59:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:59:00.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_48 snapshot 2024-09-11T08:59:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:59:01.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_48 2024-09-11T08:59:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_48': 2024-09-11T08:59:01.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14b1979412e4 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14b1979412e4 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:59:00 2024 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:59:00 2024 2024-09-11T08:59:01.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:59:00 2024 2024-09-11T08:59:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:59:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:59:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: ef84862e-99c3-4940-9190-5144ad96e521 2024-09-11T08:59:01.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:59:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:59:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_48 2024-09-11T08:59:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T08:59:01.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:59:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T08:59:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:59:01.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_48 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:59:01.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 50271232 bytes 2024-09-11T08:59:02.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 149089280 bytes 2024-09-11T08:59:02.407 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 163256320 bytes 2024-09-11T08:59:02.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 175375360 bytes 2024-09-11T08:59:02.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 245207040 bytes 2024-09-11T08:59:02.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 277131264 bytes 2024-09-11T08:59:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 297172992 bytes 2024-09-11T08:59:03.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 342427648 bytes 2024-09-11T08:59:03.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 362202112 bytes 2024-09-11T08:59:03.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 381173760 bytes 2024-09-11T08:59:03.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 407745536 bytes 2024-09-11T08:59:03.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 440577024 bytes 2024-09-11T08:59:04.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 529920000 bytes 2024-09-11T08:59:04.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 550587392 bytes 2024-09-11T08:59:04.359 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 575482880 bytes 2024-09-11T08:59:04.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 615541760 bytes 2024-09-11T08:59:04.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 635049984 bytes 2024-09-11T08:59:04.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 647479296 bytes 2024-09-11T08:59:05.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 724335616 bytes 2024-09-11T08:59:05.268 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 771436544 bytes 2024-09-11T08:59:05.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 772051968 bytes 2024-09-11T08:59:05.524 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 815922176 bytes 2024-09-11T08:59:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 843238400 bytes 2024-09-11T08:59:05.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 862090240 bytes 2024-09-11T08:59:06.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 927931392 bytes 2024-09-11T08:59:06.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 945522688 bytes 2024-09-11T08:59:06.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 959090688 bytes 2024-09-11T08:59:06.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 966087680 bytes 2024-09-11T08:59:06.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1004453888 bytes 2024-09-11T08:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:06.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_49 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_49 snapshot 128M 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_49 128M 2024-09-11T08:59:06.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:06.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_49 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_49 snapshot 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:59:06.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_49 snapshot 2024-09-11T08:59:07.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:59:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_49 2024-09-11T08:59:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_49': 2024-09-11T08:59:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:59:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:59:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:59:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14bd90b2839d 2024-09-11T08:59:07.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14bd90b2839d 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:59:06 2024 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:59:06 2024 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:59:06 2024 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: ef6d8f68-5e32-4505-b07d-5119f8fb90c8 2024-09-11T08:59:07.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:59:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:59:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_49 2024-09-11T08:59:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:07.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T08:59:07.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:59:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T08:59:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:59:07.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_49 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:59:08.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 114432000 bytes 2024-09-11T08:59:08.386 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T08:59:08.386 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T08:59:08.386 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T08:59:08.386 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T08:59:08.387 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T08:59:08.389 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T08:59:08.389 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-11T08:59:08.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 218604544 bytes 2024-09-11T08:59:08.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 266215424 bytes 2024-09-11T08:59:08.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 306457600 bytes 2024-09-11T08:59:09.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 352762880 bytes 2024-09-11T08:59:09.213 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 406695936 bytes 2024-09-11T08:59:09.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 459805696 bytes 2024-09-11T08:59:09.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 509800448 bytes 2024-09-11T08:59:09.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 510433280 bytes 2024-09-11T08:59:09.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 573629440 bytes 2024-09-11T08:59:10.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 680976384 bytes 2024-09-11T08:59:10.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 731564032 bytes 2024-09-11T08:59:10.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 761099264 bytes 2024-09-11T08:59:10.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 785804288 bytes 2024-09-11T08:59:10.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 818220032 bytes 2024-09-11T08:59:11.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 859277312 bytes 2024-09-11T08:59:11.270 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 901068800 bytes 2024-09-11T08:59:11.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 963427328 bytes 2024-09-11T08:59:11.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1005924352 bytes 2024-09-11T08:59:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1050636288 bytes 2024-09-11T08:59:12.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1101495296 bytes 2024-09-11T08:59:12.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1135070208 bytes 2024-09-11T08:59:12.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1158586368 bytes 2024-09-11T08:59:12.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1223141376 bytes 2024-09-11T08:59:12.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1305806848 bytes 2024-09-11T08:59:12.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1327963136 bytes 2024-09-11T08:59:13.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1373114368 bytes 2024-09-11T08:59:13.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1417109504 bytes 2024-09-11T08:59:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1459364864 bytes 2024-09-11T08:59:13.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1565984768 bytes 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_50 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image_and_enable_mirror cluster2 mirror image_50 snapshot 128M 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:13.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n snapshot ']' 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_50 128M 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T08:59:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T08:59:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 128M ']' 2024-09-11T08:59:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=128M 2024-09-11T08:59:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T08:59:13.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 128M --image-feature layering,exclusive-lock mirror/image_50 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ is_pool_mirror_mode_image mirror 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' image = image ']' 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_50 snapshot 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T08:59:13.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_50 snapshot 2024-09-11T08:59:14.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T08:59:14.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_50 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_50': 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 14cefae23d08 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.14cefae23d08 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:59:13 2024 2024-09-11T08:59:14.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:59:13 2024 2024-09-11T08:59:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:59:13 2024 2024-09-11T08:59:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T08:59:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T08:59:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 721a420b-6433-4096-afbf-22d3debada44 2024-09-11T08:59:14.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T08:59:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n '' ']' 2024-09-11T08:59:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_50 2024-09-11T08:59:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T08:59:14.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T08:59:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=11 2024-09-11T08:59:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T08:59:14.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 11s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_50 --debug-rbd=20 --debug-journaler=20 2024-09-11T08:59:15.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 55759872 bytes 2024-09-11T08:59:15.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 79703040 bytes 2024-09-11T08:59:15.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 127459328 bytes 2024-09-11T08:59:15.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 221882368 bytes 2024-09-11T08:59:15.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 250973184 bytes 2024-09-11T08:59:16.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 311155712 bytes 2024-09-11T08:59:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 326981632 bytes 2024-09-11T08:59:16.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 373895168 bytes 2024-09-11T08:59:16.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 418925568 bytes 2024-09-11T08:59:16.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 472046592 bytes 2024-09-11T08:59:17.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 544026624 bytes 2024-09-11T08:59:17.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 586863616 bytes 2024-09-11T08:59:17.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 669926400 bytes 2024-09-11T08:59:17.704 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 690883584 bytes 2024-09-11T08:59:17.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 721438720 bytes 2024-09-11T08:59:17.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 723805184 bytes 2024-09-11T08:59:18.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 784635904 bytes 2024-09-11T08:59:18.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 815296512 bytes 2024-09-11T08:59:18.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 887900160 bytes 2024-09-11T08:59:18.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 916600832 bytes 2024-09-11T08:59:18.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 929430528 bytes 2024-09-11T08:59:19.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 993267712 bytes 2024-09-11T08:59:19.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1057018880 bytes 2024-09-11T08:59:19.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1101138944 bytes 2024-09-11T08:59:19.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1154305024 bytes 2024-09-11T08:59:20.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1257810944 bytes 2024-09-11T08:59:20.259 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1307551744 bytes 2024-09-11T08:59:20.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1326112768 bytes 2024-09-11T08:59:20.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1344615424 bytes 2024-09-11T08:59:20.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1407041536 bytes 2024-09-11T08:59:20.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1415908352 bytes 2024-09-11T08:59:21.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1447346176 bytes 2024-09-11T08:59:21.243 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1486423040 bytes 2024-09-11T08:59:21.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1507702784 bytes 2024-09-11T08:59:21.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1508312064 bytes 2024-09-11T08:59:21.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1538539520 bytes 2024-09-11T08:59:21.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1597235200 bytes 2024-09-11T08:59:22.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1632415744 bytes 2024-09-11T08:59:22.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1678279680 bytes 2024-09-11T08:59:22.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1735943168 bytes 2024-09-11T08:59:22.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1830735872 bytes 2024-09-11T08:59:22.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1907453952 bytes 2024-09-11T08:59:23.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1915709440 bytes 2024-09-11T08:59:23.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1958866944 bytes 2024-09-11T08:59:23.525 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2022892544 bytes 2024-09-11T08:59:23.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2042816512 bytes 2024-09-11T08:59:23.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2101556224 bytes 2024-09-11T08:59:24.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2149463040 bytes 2024-09-11T08:59:24.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2243548160 bytes 2024-09-11T08:59:24.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2282332160 bytes 2024-09-11T08:59:24.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2297513984 bytes 2024-09-11T08:59:24.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2358019072 bytes 2024-09-11T08:59:25.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2411265024 bytes 2024-09-11T08:59:25.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2459672576 bytes 2024-09-11T08:59:25.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2479649792 bytes 2024-09-11T08:59:25.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2562331648 bytes 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_pool_images cluster2 mirror 50 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_count=50 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:59:25.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local count 2024-09-11T08:59:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_count=0 2024-09-11T08:59:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:59:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 40 2024-09-11T08:59:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in `seq 1 40` 2024-09-11T08:59:25.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 1 -ne 1 2024-09-11T08:59:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T08:59:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep 'images: ' 2024-09-11T08:59:25.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cut '-d ' -f 2 2024-09-11T08:59:25.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T08:59:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ count=50 2024-09-11T08:59:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 50 = 50 2024-09-11T08:59:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:25.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_pool_healthy cluster2 mirror 2024-09-11T08:59:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:59:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state 2024-09-11T08:59:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 40 2024-09-11T08:59:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in `seq 1 40` 2024-09-11T08:59:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 1 -ne 1 2024-09-11T08:59:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T08:59:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep 'image health:' 2024-09-11T08:59:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cut '-d ' -f 3 2024-09-11T08:59:25.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T08:59:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=WARNING 2024-09-11T08:59:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test WARNING = ERROR 2024-09-11T08:59:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test WARNING = OK 2024-09-11T08:59:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in `seq 1 40` 2024-09-11T08:59:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 2 -ne 1 2024-09-11T08:59:25.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 30 2024-09-11T08:59:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster2 -p mirror mirror pool status 2024-09-11T08:59:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep 'image health:' 2024-09-11T08:59:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cut '-d ' -f 3 2024-09-11T08:59:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T08:59:55.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=OK 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test OK = ERROR 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test OK = OK 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_pool_images cluster1 mirror 50 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_count=50 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local count 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_count=0 2024-09-11T08:59:55.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:59:55.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 40 2024-09-11T08:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in `seq 1 40` 2024-09-11T08:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 1 -ne 1 2024-09-11T08:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T08:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep 'images: ' 2024-09-11T08:59:55.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cut '-d ' -f 2 2024-09-11T08:59:55.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ count=50 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 50 = 50 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_pool_healthy cluster1 mirror 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state 2024-09-11T08:59:56.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 40 2024-09-11T08:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in `seq 1 40` 2024-09-11T08:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 1 -ne 1 2024-09-11T08:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T08:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep 'image health:' 2024-09-11T08:59:56.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cut '-d ' -f 3 2024-09-11T08:59:56.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T08:59:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=OK 2024-09-11T08:59:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test OK = ERROR 2024-09-11T08:59:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test OK = OK 2024-09-11T08:59:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ testlog 'TEST: compare many images' 2024-09-11T08:59:56.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ tee -a /tmp/tmp.LFCa6yxq8J/rbd-mirror.test.log 2024-09-11T08:59:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ date '+%F %T' 2024-09-11T08:59:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo 2024-09-11 08:59:56 TEST: compare many images 2024-09-11T08:59:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:2024-09-11 08:59:56 TEST: compare many images 2024-09-11T08:59:56.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 50 2024-09-11T08:59:56.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T08:59:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_1 2024-09-11T08:59:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_1 snap 2024-09-11T08:59:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T08:59:56.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_1@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T08:59:57.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_1 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_1 started 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:59:57.852 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T08:59:58.283 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.259354342419025) > weight (0.25) 2024-09-11T08:59:58.283 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T08:59:58.284 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T08:59:58.284 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-11T08:59:58.285 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T08:59:58.285+0000 7fe75a7f6640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 43565) UID: 1000 2024-09-11T08:59:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_1 started 2024-09-11T08:59:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T08:59:58.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T08:59:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T08:59:58.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T08:59:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_1 2024-09-11T08:59:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T08:59:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T08:59:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:59:58.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T08:59:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T08:59:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T08:59:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T08:59:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T08:59:58.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T08:59:58.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_1 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T08:59:58.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T08:59:58.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T08:59:58.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T08:59:58.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:59:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:59:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T08:59:58.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_1 2024-09-11T08:59:59.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T08:59:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T08:59:59.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T08:59:59.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T08:59:59.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T08:59:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T08:59:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T08:59:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T08:59:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_1 2024-09-11T08:59:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:59:59.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_1 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_1.status 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_1.status 2024-09-11T08:59:59.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:59:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.status 2024-09-11T08:59:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_1.status 2024-09-11T08:59:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_1.status 2024-09-11T08:59:59.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T08:59:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.status 2024-09-11T08:59:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_1 2024-09-11T08:59:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:59.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_1 2024-09-11T08:59:59.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 86 2024-09-11T08:59:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_1 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.status 2024-09-11T08:59:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T08:59:59.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T08:59:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T08:59:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.status 2024-09-11T08:59:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_1 --format xml 2024-09-11T08:59:59.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T08:59:59.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T08:59:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=86 2024-09-11T08:59:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T08:59:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T08:59:59.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:00:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.status 2024-09-11T09:00:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:00:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.status 2024-09-11T09:00:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-11T09:00:00.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:00.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:00.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 86 = '' 2024-09-11T09:00:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:00.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.status 2024-09-11T09:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:00:00.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.status 2024-09-11T09:00:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-11T09:00:00.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:00.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 86 = '' 2024-09-11T09:00:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:00.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_1 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.status 2024-09-11T09:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.status 2024-09-11T09:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_1 --format xml 2024-09-11T09:00:01.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:01.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=86 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 86 = 86 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_1 snap 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:01.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:00:01.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:02.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_1@snap 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_1': 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10adaf558382 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10adaf558382 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:00:02.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:14 2024 2024-09-11T09:00:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:14 2024 2024-09-11T09:00:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:14 2024 2024-09-11T09:00:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:00:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:00:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:00:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 209601d8-0ce8-4bab-bfd1-bea5e9278d31 2024-09-11T09:00:02.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_1 snap 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.export 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.export 2024-09-11T09:00:02.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.export 2024-09-11T09:00:02.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_1@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.export 2024-09-11T09:00:02.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:02.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_1@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.export 2024-09-11T09:00:03.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:03.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.export 2024-09-11T09:00:03.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_1.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_1.export 2024-09-11T09:00:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:03.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:00:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_2 2024-09-11T09:00:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_2 snap 2024-09-11T09:00:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:03.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_2@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_2 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_2 started 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:04.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:00:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:04.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:05.285 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T09:00:05.285 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:00:05.285 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T09:00:05.286 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T09:00:05.286 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T09:00:05.289 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T09:00:05.289 INFO:tasks.rbd_mirror_thrash:waiting for 76 secs before thrashing 2024-09-11T09:00:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_2 started 2024-09-11T09:00:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:00:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:00:05.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:00:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:00:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_2 2024-09-11T09:00:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:00:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:00:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:00:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:00:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:00:05.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:05.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_2 2024-09-11T09:00:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:05.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:05.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:05.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:05.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:05.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:05.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_2 2024-09-11T09:00:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 2 closest matches: 2024-09-11T09:00:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_20 2024-09-11T09:00:06.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_23 2024-09-11T09:00:06.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_2 2024-09-11T09:00:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:06.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:06.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:06.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:06.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:06.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:06.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_2 2024-09-11T09:00:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 3 closest matches: 2024-09-11T09:00:06.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_25 2024-09-11T09:00:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_27 2024-09-11T09:00:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_29 2024-09-11T09:00:06.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_2 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:06.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:06.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:06.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:06.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:06.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_2 2024-09-11T09:00:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 5 closest matches: 2024-09-11T09:00:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_21 2024-09-11T09:00:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_22 2024-09-11T09:00:06.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_24 2024-09-11T09:00:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_26 2024-09-11T09:00:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_28 2024-09-11T09:00:06.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:06.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_2 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:06.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:00:06.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:00:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:00:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:00:06.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:00:06.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:00:06.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:06.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_2 2024-09-11T09:00:06.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:00:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:00:06.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:00:06.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:00:06.493 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:00:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:00:06.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_2 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_2 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:06.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_2.status 2024-09-11T09:00:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_2.status 2024-09-11T09:00:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.status 2024-09-11T09:00:06.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_2.status 2024-09-11T09:00:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_2.status 2024-09-11T09:00:06.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.status 2024-09-11T09:00:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_2 2024-09-11T09:00:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:06.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_2 2024-09-11T09:00:06.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 88 2024-09-11T09:00:06.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_2 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.status 2024-09-11T09:00:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.status 2024-09-11T09:00:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_2 --format xml 2024-09-11T09:00:06.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:06.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:00:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=88 2024-09-11T09:00:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:06.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:00:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.status 2024-09-11T09:00:07.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.status 2024-09-11T09:00:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-11T09:00:07.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:07.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 88 = '' 2024-09-11T09:00:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:07.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:00:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.status 2024-09-11T09:00:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:07.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.status 2024-09-11T09:00:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-11T09:00:07.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:07.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:07.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:07.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 88 = '' 2024-09-11T09:00:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:07.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:00:08.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_2 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.status 2024-09-11T09:00:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.status 2024-09-11T09:00:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_2 --format xml 2024-09-11T09:00:08.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:08.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=88 2024-09-11T09:00:08.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 88 = 88 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_2 snap 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:00:08.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:09.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_2@snap 2024-09-11T09:00:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_2': 2024-09-11T09:00:09.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10b33dd47c55 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10b33dd47c55 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:21 2024 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:21 2024 2024-09-11T09:00:09.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:21 2024 2024-09-11T09:00:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:00:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:00:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:00:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 6176f630-2f7a-4485-9979-55a0a1132ec3 2024-09-11T09:00:09.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:00:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_2 snap 2024-09-11T09:00:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:09.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:00:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:00:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.export 2024-09-11T09:00:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.export 2024-09-11T09:00:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.export 2024-09-11T09:00:09.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_2@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.export 2024-09-11T09:00:09.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:09.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_2@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.export 2024-09-11T09:00:10.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:10.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.export 2024-09-11T09:00:10.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_2.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_2.export 2024-09-11T09:00:10.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_3 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_3 snap 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:10.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_3@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:00:11.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_3 2024-09-11T09:00:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_3 started 2024-09-11T09:00:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:11.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:00:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:11.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_3 started 2024-09-11T09:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:00:12.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:00:12.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_3 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:00:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_3 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:12.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_3 2024-09-11T09:00:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 1 closest matches: 2024-09-11T09:00:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_39 2024-09-11T09:00:13.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_3 2024-09-11T09:00:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:13.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:13.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:13.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:13.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:13.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_3 2024-09-11T09:00:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 4 closest matches: 2024-09-11T09:00:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_33 2024-09-11T09:00:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_34 2024-09-11T09:00:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_37 2024-09-11T09:00:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_38 2024-09-11T09:00:13.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:13.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_3 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:13.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:13.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:13.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:13.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:13.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:13.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_3 2024-09-11T09:00:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 1 closest matches: 2024-09-11T09:00:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_35 2024-09-11T09:00:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:13.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_3 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:13.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:00:13.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:13.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:00:13.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:00:13.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:13.364 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_3 2024-09-11T09:00:13.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:00:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:00:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:00:13.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:00:13.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:00:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:00:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:00:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:13.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_3 2024-09-11T09:00:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:00:13.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:00:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_3 2024-09-11T09:00:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:13.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_3.status 2024-09-11T09:00:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_3.status 2024-09-11T09:00:13.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:13.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.status 2024-09-11T09:00:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_3.status 2024-09-11T09:00:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_3.status 2024-09-11T09:00:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_3 2024-09-11T09:00:13.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:13.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_3 2024-09-11T09:00:13.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 90 2024-09-11T09:00:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_3 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.status 2024-09-11T09:00:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:13.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.status 2024-09-11T09:00:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_3 --format xml 2024-09-11T09:00:13.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:14.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:00:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=90 2024-09-11T09:00:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:14.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:14.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:00:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:14.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T09:00:14.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:14.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=37 2024-09-11T09:00:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 90 = 37 2024-09-11T09:00:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:14.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:00:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T09:00:14.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:14.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=37 2024-09-11T09:00:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 90 = 37 2024-09-11T09:00:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:00:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:15.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:15.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T09:00:15.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:15.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:15.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 90 = '' 2024-09-11T09:00:15.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:15.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:00:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_3 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:17.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.status 2024-09-11T09:00:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_3 --format xml 2024-09-11T09:00:17.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:17.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=90 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 90 = 90 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_3 snap 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:17.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:00:17.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:18.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_3@snap 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_3': 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 118f10e419d4 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.118f10e419d4 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:00:18.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:35 2024 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:35 2024 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:35 2024 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8efd8144-a459-4554-b275-2ec378bad1af 2024-09-11T09:00:18.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:00:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_3 snap 2024-09-11T09:00:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:00:18.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:00:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.export 2024-09-11T09:00:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.export 2024-09-11T09:00:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.export 2024-09-11T09:00:18.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_3@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.export 2024-09-11T09:00:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:18.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_3@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.export 2024-09-11T09:00:18.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:18.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.export 2024-09-11T09:00:19.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_3.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_3.export 2024-09-11T09:00:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:00:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_4 2024-09-11T09:00:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_4 snap 2024-09-11T09:00:19.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:19.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_4@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:00:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_4 2024-09-11T09:00:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:20.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_4 started 2024-09-11T09:00:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:00:20.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:20.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:21.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_4 started 2024-09-11T09:00:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:00:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:00:21.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:00:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_4 2024-09-11T09:00:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:00:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:00:21.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:00:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:00:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:00:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:00:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:21.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_4 2024-09-11T09:00:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:21.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:21.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:21.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:21.601 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_4 2024-09-11T09:00:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 1 closest matches: 2024-09-11T09:00:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_40 2024-09-11T09:00:21.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:21.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_4 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:21.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:21.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:21.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:21.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:21.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_4 2024-09-11T09:00:21.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:no valid command found; 4 closest matches: 2024-09-11T09:00:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_41 2024-09-11T09:00:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_44 2024-09-11T09:00:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_45 2024-09-11T09:00:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status mirror/image_49 2024-09-11T09:00:21.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_4 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:21.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:21.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:21.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:21.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:21.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_4 2024-09-11T09:00:21.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:00:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:00:21.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:00:22.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:00:22.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_4 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:22.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_4 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_4.status 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_4.status 2024-09-11T09:00:22.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.status 2024-09-11T09:00:22.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_4.status 2024-09-11T09:00:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_4.status 2024-09-11T09:00:22.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:22.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_4 2024-09-11T09:00:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:22.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_4 2024-09-11T09:00:22.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 92 2024-09-11T09:00:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_4 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.status 2024-09-11T09:00:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:22.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.status 2024-09-11T09:00:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_4 --format xml 2024-09-11T09:00:22.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:22.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:00:22.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=92 2024-09-11T09:00:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:22.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:00:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:22.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T09:00:22.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:22.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=38 2024-09-11T09:00:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 92 = 38 2024-09-11T09:00:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:22.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:23.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T09:00:23.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:23.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 92 = '' 2024-09-11T09:00:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:23.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:00:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:24.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T09:00:24.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:24.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:24.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:24.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 92 = '' 2024-09-11T09:00:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:24.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:00:24.390 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T09:00:24.390 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T09:00:24.390 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T09:00:24.391 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:00:24.392+0000 7f1035a04640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 43903) UID: 1000 2024-09-11T09:00:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_4 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:25.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.status 2024-09-11T09:00:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_4 --format xml 2024-09-11T09:00:25.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:25.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=92 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 92 = 92 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_4 snap 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:00:25.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:26.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_4@snap 2024-09-11T09:00:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_4': 2024-09-11T09:00:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:00:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:00:26.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 127c887e1613 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.127c887e1613 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:50 2024 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:50 2024 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:50 2024 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:00:26.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 39633655-9b79-45e6-ac71-541b46d2ea10 2024-09-11T09:00:26.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:00:27.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_4 snap 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.export 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.export 2024-09-11T09:00:27.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.export 2024-09-11T09:00:27.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_4@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.export 2024-09-11T09:00:27.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:27.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_4@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.export 2024-09-11T09:00:27.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:00:27.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.export 2024-09-11T09:00:27.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_4.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_4.export 2024-09-11T09:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_5 2024-09-11T09:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_5 snap 2024-09-11T09:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:27.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:00:27.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_5@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:00:28.391 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T09:00:28.391 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:00:28.392 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T09:00:28.392 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T09:00:28.392 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T09:00:28.395 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T09:00:28.395 INFO:tasks.rbd_mirror_thrash:waiting for 73 secs before thrashing 2024-09-11T09:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_5 2024-09-11T09:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:29.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_5 started 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:29.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:00:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_5 started 2024-09-11T09:00:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:30.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:00:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:00:30.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_5 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:00:30.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:00:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:30.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:30.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:30.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:30.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:30.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:30.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:30.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T09:00:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:30.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-11T09:00:30.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:30.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:30.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:30.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:30.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:30.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-11T09:00:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:30.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:30.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:30.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:30.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:30.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-11T09:00:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:31.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:31.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-09-11T09:00:31.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:00:31.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:31.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:00:31.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:00:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:31.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-09-11T09:00:31.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:31.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 1 2024-09-11T09:00:32.213 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:32.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T09:00:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:32.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:32.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:32.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:32.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:32.244 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:32.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T09:00:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:32.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-11T09:00:32.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:32.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:32.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:32.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:32.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:32.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:32.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-11T09:00:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:32.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-11T09:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:32.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.492 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:32.493 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:32.494 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:32.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:32.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:32.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-11T09:00:32.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:32.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:00:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:00:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:32.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:32.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-09-11T09:00:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:32.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:32.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 2 2024-09-11T09:00:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:34.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:34.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:34.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:34.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:34.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T09:00:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:34.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:34.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:34.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:34.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:34.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-11T09:00:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:35.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:35.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:35.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:35.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:35.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:35.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:35.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-11T09:00:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:35.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:00:35.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:00:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:00:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:00:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:00:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:00:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:00:35.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:00:35.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:35.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-09-11T09:00:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:35.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:35.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 4 2024-09-11T09:00:39.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:39.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:39.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:39.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:39.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:39.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:39.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:39.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T09:00:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:39.461 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:39.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:39.464 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:39.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:39.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:39.487 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:39.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-11T09:00:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:39.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-11T09:00:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:39.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:39.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:39.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:39.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:39.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:39.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-11T09:00:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:39.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:39.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:00:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:00:39.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:39.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:00:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:00:39.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:00:39.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:39.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-09-11T09:00:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:39.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:39.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 8 2024-09-11T09:00:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:47.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:47.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:47.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:47.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T09:00:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:48.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:48.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:48.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:48.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:48.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:48.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:48.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-11T09:00:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:48.161 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-11T09:00:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:48.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:48.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:48.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:48.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:48.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:48.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:48.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-11T09:00:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:48.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_5 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:48.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:00:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:00:48.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:00:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:00:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:48.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:00:48.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:00:48.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_5 2024-09-11T09:00:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:48.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:00:48.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 8 2024-09-11T09:00:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:00:56.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_5 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:56.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:00:56.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:00:56.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:00:56.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:00:56.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:00:56.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_5 2024-09-11T09:00:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:56.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:56.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_5 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:00:56.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:56.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:00:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:00:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:00:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:00:56.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:00:56.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_5 2024-09-11T09:00:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_5'] 2024-09-11T09:00:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:00:56.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:00:56.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_5 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:00:56.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:00:56.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:00:56.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:00:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:00:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:00:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:00:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:00:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:00:56.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:00:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:00:56.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_5 2024-09-11T09:00:56.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:00:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:00:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:00:56.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:00:56.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:00:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:00:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:00:56.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_5 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_5 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:56.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_5.status 2024-09-11T09:00:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_5.status 2024-09-11T09:00:56.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:56.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.status 2024-09-11T09:00:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_5.status 2024-09-11T09:00:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_5.status 2024-09-11T09:00:56.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:00:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:00:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_5 2024-09-11T09:00:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:56.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_5 2024-09-11T09:00:57.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 94 2024-09-11T09:00:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_5 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.status 2024-09-11T09:00:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:00:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.status 2024-09-11T09:00:57.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_5 --format xml 2024-09-11T09:00:57.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:57.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:00:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=94 2024-09-11T09:00:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:57.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:00:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:00:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:00:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T09:00:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:58.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:58.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=39 2024-09-11T09:00:58.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 94 = 39 2024-09-11T09:00:58.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:58.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:58.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:00:58.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T09:00:58.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:58.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:58.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 94 = '' 2024-09-11T09:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:58.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:00:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:00:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:00:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:00:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:00:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:00:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T09:00:59.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:00:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:00:59.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:00:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:00:59.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:00:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:00:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:00:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 94 = '' 2024-09-11T09:00:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:00:59.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:01:00.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_5 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:01:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:01:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.status 2024-09-11T09:01:01.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_5 --format xml 2024-09-11T09:01:01.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:01.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=94 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 94 = 94 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_5 snap 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:01.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:02.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_5@snap 2024-09-11T09:01:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_5': 2024-09-11T09:01:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:02.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10adda154fd2 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10adda154fd2 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:51:56 2024 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:51:56 2024 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:51:56 2024 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 57b1ae5d-d195-4c6d-b9e5-3241f71e3de5 2024-09-11T09:01:02.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_5 snap 2024-09-11T09:01:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:01:02.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.export 2024-09-11T09:01:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.export 2024-09-11T09:01:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.export 2024-09-11T09:01:02.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_5@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.export 2024-09-11T09:01:02.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:02.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_5@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.export 2024-09-11T09:01:02.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:02.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.export 2024-09-11T09:01:02.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_5.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_5.export 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_6 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_6 snap 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:02.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_6@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:04.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_6 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_6 started 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:04.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_6 started 2024-09-11T09:01:05.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:05.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_6 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:05.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:05.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_6 2024-09-11T09:01:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:05.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:05.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:05.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:05.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:05.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:05.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:05.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:05.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_6 2024-09-11T09:01:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-11T09:01:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:05.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_6 2024-09-11T09:01:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:01:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:05.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:01:05.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:01:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:01:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:05.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:01:05.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:01:05.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:01:05.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:05.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_6 2024-09-11T09:01:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_6'] 2024-09-11T09:01:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:06.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_6 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:01:06.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:01:06.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:06.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:01:06.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:01:06.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:06.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_6 2024-09-11T09:01:06.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:06.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:06.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:06.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_6 2024-09-11T09:01:06.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_6 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_6.status 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_6.status 2024-09-11T09:01:06.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:06.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.status 2024-09-11T09:01:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_6.status 2024-09-11T09:01:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_6.status 2024-09-11T09:01:06.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.status 2024-09-11T09:01:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_6 2024-09-11T09:01:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:06.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_6 2024-09-11T09:01:06.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 96 2024-09-11T09:01:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_6 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.status 2024-09-11T09:01:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:06.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.status 2024-09-11T09:01:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_6 --format xml 2024-09-11T09:01:06.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:06.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=96 2024-09-11T09:01:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:06.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.status 2024-09-11T09:01:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:06.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.status 2024-09-11T09:01:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-11T09:01:06.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:07.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:07.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 96 = '' 2024-09-11T09:01:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:07.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.status 2024-09-11T09:01:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:07.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.status 2024-09-11T09:01:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-11T09:01:07.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:07.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:07.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 96 = '' 2024-09-11T09:01:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:07.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_6 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.status 2024-09-11T09:01:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:08.343 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.status 2024-09-11T09:01:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_6 --format xml 2024-09-11T09:01:08.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:08.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=96 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 96 = 96 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_6 snap 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:08.415 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:09.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_6@snap 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_6': 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10b3d0375602 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10b3d0375602 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:09.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:04 2024 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:04 2024 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:04 2024 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: bbfebdd5-c61c-4f17-9e2f-1d0a11cb2d73 2024-09-11T09:01:09.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_6 snap 2024-09-11T09:01:09.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.export 2024-09-11T09:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.export 2024-09-11T09:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.export 2024-09-11T09:01:09.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_6@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.export 2024-09-11T09:01:09.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:09.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_6@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.export 2024-09-11T09:01:10.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:10.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.export 2024-09-11T09:01:10.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_6.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_6.export 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_7 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_7 snap 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:10.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_7@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:11.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_7 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_7 started 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:11.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_7 started 2024-09-11T09:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:12.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_7 2024-09-11T09:01:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:12.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_7 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:12.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:12.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:12.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:12.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:12.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:12.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_7 2024-09-11T09:01:12.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:12.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:12.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_7 2024-09-11T09:01:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:12.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_7 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_7.status 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_7.status 2024-09-11T09:01:12.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.status 2024-09-11T09:01:12.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_7.status 2024-09-11T09:01:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_7.status 2024-09-11T09:01:12.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:12.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.status 2024-09-11T09:01:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_7 2024-09-11T09:01:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:12.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_7 2024-09-11T09:01:13.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 98 2024-09-11T09:01:13.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_7 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.status 2024-09-11T09:01:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.status 2024-09-11T09:01:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_7 --format xml 2024-09-11T09:01:13.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:13.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:13.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=98 2024-09-11T09:01:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:13.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.status 2024-09-11T09:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:14.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.status 2024-09-11T09:01:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-11T09:01:14.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:14.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:14.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 98 = '' 2024-09-11T09:01:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:14.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.status 2024-09-11T09:01:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:14.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.status 2024-09-11T09:01:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-11T09:01:14.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:14.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:14.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:14.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 98 = '' 2024-09-11T09:01:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:14.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_7 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.status 2024-09-11T09:01:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:15.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.status 2024-09-11T09:01:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_7 --format xml 2024-09-11T09:01:15.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:15.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:15.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=98 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 98 = 98 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_7 snap 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:15.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:16.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_7@snap 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_7': 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 118f6065fb4a 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.118f6065fb4a 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:16.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:15 2024 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:15 2024 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:15 2024 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 75596a83-10f1-4277-ad91-67db500695a7 2024-09-11T09:01:16.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_7 snap 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.export 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.export 2024-09-11T09:01:16.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.export 2024-09-11T09:01:16.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_7@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.export 2024-09-11T09:01:16.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:16.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_7@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.export 2024-09-11T09:01:17.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:17.268 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.export 2024-09-11T09:01:17.346 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_7.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_7.export 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_8 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_8 snap 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:17.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_8@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_8 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_8 started 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:18.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:18.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_8 started 2024-09-11T09:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:19.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_8 2024-09-11T09:01:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:19.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:19.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:19.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:19.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_8 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:19.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:19.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:19.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:19.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:19.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_8 2024-09-11T09:01:19.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:19.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:19.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:19.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_8 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_8 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:19.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_8.status 2024-09-11T09:01:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_8.status 2024-09-11T09:01:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.status 2024-09-11T09:01:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_8.status 2024-09-11T09:01:19.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_8.status 2024-09-11T09:01:19.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.status 2024-09-11T09:01:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_8 2024-09-11T09:01:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:19.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_8 2024-09-11T09:01:20.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 100 2024-09-11T09:01:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_8 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.status 2024-09-11T09:01:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:20.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.status 2024-09-11T09:01:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_8 --format xml 2024-09-11T09:01:20.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:20.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=100 2024-09-11T09:01:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:20.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.status 2024-09-11T09:01:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:21.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.status 2024-09-11T09:01:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:21.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-11T09:01:21.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:21.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:21.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 100 = '' 2024-09-11T09:01:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:21.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:21.291 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.9947090340092907) > weight (0.25) 2024-09-11T09:01:21.291 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.2750182023547654) > weight (0.25) 2024-09-11T09:01:21.291 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5018552773762615) > weight (0.25) 2024-09-11T09:01:21.291 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.6432913093863191) > weight (0.25) 2024-09-11T09:01:21.291 INFO:tasks.rbd_mirror_thrash:waiting for 66 secs before thrashing 2024-09-11T09:01:21.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.status 2024-09-11T09:01:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.status 2024-09-11T09:01:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-11T09:01:21.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:21.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:21.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:21.611 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 100 = '' 2024-09-11T09:01:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:21.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_8 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.status 2024-09-11T09:01:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:22.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.status 2024-09-11T09:01:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_8 --format xml 2024-09-11T09:01:22.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:22.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:22.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=100 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 100 = 100 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_8 snap 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:22.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:23.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_8@snap 2024-09-11T09:01:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_8': 2024-09-11T09:01:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 127cc2d7dde 2024-09-11T09:01:23.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.127cc2d7dde 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:21 2024 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:21 2024 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:21 2024 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 880a8fca-463b-4b43-b6d1-4b6ea06390b7 2024-09-11T09:01:23.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_8 snap 2024-09-11T09:01:23.557 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:01:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.export 2024-09-11T09:01:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.export 2024-09-11T09:01:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.export 2024-09-11T09:01:23.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_8@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.export 2024-09-11T09:01:23.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:23.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_8@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.export 2024-09-11T09:01:24.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:24.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.export 2024-09-11T09:01:24.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_8.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_8.export 2024-09-11T09:01:24.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_9 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_9 snap 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:24.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_9@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_9 2024-09-11T09:01:25.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_9 started 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:25.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_9 started 2024-09-11T09:01:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:26.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:26.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_9 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:26.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:26.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_9 2024-09-11T09:01:26.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:26.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:26.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:26.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:26.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:26.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_9 2024-09-11T09:01:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_9'] 2024-09-11T09:01:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:26.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_9 2024-09-11T09:01:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:01:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:01:26.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:01:26.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:01:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:01:26.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:01:26.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:01:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:26.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_9 2024-09-11T09:01:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:27.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:27.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:27.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:27.111 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_9 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_9 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:27.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_9.status 2024-09-11T09:01:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_9.status 2024-09-11T09:01:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:27.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.status 2024-09-11T09:01:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_9.status 2024-09-11T09:01:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_9.status 2024-09-11T09:01:27.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.status 2024-09-11T09:01:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_9 2024-09-11T09:01:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:27.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_9 2024-09-11T09:01:27.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 102 2024-09-11T09:01:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_9 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.status 2024-09-11T09:01:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:27.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.status 2024-09-11T09:01:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_9 --format xml 2024-09-11T09:01:27.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:27.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=102 2024-09-11T09:01:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:27.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:28.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.status 2024-09-11T09:01:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.status 2024-09-11T09:01:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-11T09:01:28.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:28.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:28.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 102 = '' 2024-09-11T09:01:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:28.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.status 2024-09-11T09:01:28.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.status 2024-09-11T09:01:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-11T09:01:28.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:28.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:28.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 102 = '' 2024-09-11T09:01:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:28.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_9 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.status 2024-09-11T09:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.status 2024-09-11T09:01:29.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_9 --format xml 2024-09-11T09:01:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:29.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:29.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=102 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 102 = 102 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_9 snap 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:29.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:30.505 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_9@snap 2024-09-11T09:01:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_9': 2024-09-11T09:01:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:30.564 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10adb285739 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10adb285739 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:30 2024 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:30 2024 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:30 2024 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 923d86f9-42b1-48a1-b1b7-848b1ff22bf5 2024-09-11T09:01:30.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_9 snap 2024-09-11T09:01:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:01:30.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.export 2024-09-11T09:01:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.export 2024-09-11T09:01:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.export 2024-09-11T09:01:30.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_9@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.export 2024-09-11T09:01:30.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_9@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.export 2024-09-11T09:01:31.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:31.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.export 2024-09-11T09:01:31.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_9.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_9.export 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_10 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_10 snap 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:31.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_10@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_10 2024-09-11T09:01:32.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_10 started 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:32.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_10 started 2024-09-11T09:01:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:33.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:33.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_10 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:33.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:33.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:33.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_10 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:33.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:33.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:33.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:33.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_10 2024-09-11T09:01:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-11T09:01:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:33.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_10 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:01:33.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:01:33.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:33.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:33.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:01:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:01:33.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:01:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:34.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_10 2024-09-11T09:01:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-11T09:01:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:34.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_10 2024-09-11T09:01:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:01:34.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:01:34.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:34.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:01:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:01:34.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:01:34.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:34.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_10 2024-09-11T09:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_10'] 2024-09-11T09:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:34.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_10 2024-09-11T09:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:34.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:01:34.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:01:34.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:01:34.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:01:34.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_10 2024-09-11T09:01:34.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:34.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:34.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:34.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_10 2024-09-11T09:01:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:34.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_10 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_10.status 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:34.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_10.status 2024-09-11T09:01:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.status 2024-09-11T09:01:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_10.status 2024-09-11T09:01:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_10.status 2024-09-11T09:01:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:34.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.status 2024-09-11T09:01:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_10 2024-09-11T09:01:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:34.420 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_10 2024-09-11T09:01:34.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 104 2024-09-11T09:01:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_10 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.status 2024-09-11T09:01:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:34.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.status 2024-09-11T09:01:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_10 --format xml 2024-09-11T09:01:34.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:34.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=104 2024-09-11T09:01:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:34.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.status 2024-09-11T09:01:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:35.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.status 2024-09-11T09:01:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-11T09:01:35.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:35.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:35.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:35.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 104 = '' 2024-09-11T09:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:35.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.status 2024-09-11T09:01:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:35.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.status 2024-09-11T09:01:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-11T09:01:35.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:35.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:35.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:35.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 104 = '' 2024-09-11T09:01:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:35.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_10 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.status 2024-09-11T09:01:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:36.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.status 2024-09-11T09:01:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_10 --format xml 2024-09-11T09:01:36.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:36.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:36.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=104 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 104 = 104 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_10 snap 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:36.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:37.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_10@snap 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_10': 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10b357a86363 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10b357a86363 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:37.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:44 2024 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:44 2024 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:44 2024 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: b544cd07-41a2-4db4-b006-4225d4700d0a 2024-09-11T09:01:37.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_10 snap 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.export 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.export 2024-09-11T09:01:37.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.export 2024-09-11T09:01:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_10@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.export 2024-09-11T09:01:37.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:37.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_10@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.export 2024-09-11T09:01:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:38.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.export 2024-09-11T09:01:38.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_10.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_10.export 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_11 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_11 snap 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:38.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_11@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_11 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_11 started 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:39.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:39.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:40.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_11 started 2024-09-11T09:01:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:40.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:40.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_11 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:40.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_11 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:40.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:40.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:40.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:40.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:40.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:40.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_11 2024-09-11T09:01:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-11T09:01:41.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:41.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_11 2024-09-11T09:01:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:01:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:41.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:01:41.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:01:41.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:01:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:01:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:01:41.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:01:41.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:01:41.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_11 2024-09-11T09:01:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-11T09:01:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:41.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_11 2024-09-11T09:01:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:01:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:01:41.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:01:41.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:01:41.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:01:41.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:01:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:01:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:01:41.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:01:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:41.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:01:41.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_11 2024-09-11T09:01:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_11'] 2024-09-11T09:01:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:01:41.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_11 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:41.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:01:41.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:41.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:01:41.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:01:41.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:01:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:01:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:01:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:01:41.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_11 2024-09-11T09:01:41.396 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.5332554116629232) > weight (0.25) 2024-09-11T09:01:41.396 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.3897488384560943) > weight (0.25) 2024-09-11T09:01:41.397 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.4976217149299741) > weight (0.25) 2024-09-11T09:01:41.397 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9952472054653214) > weight (0.25) 2024-09-11T09:01:41.397 INFO:tasks.rbd_mirror_thrash:waiting for 78 secs before thrashing 2024-09-11T09:01:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:41.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:41.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:41.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_11 2024-09-11T09:01:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_11 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:41.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_11.status 2024-09-11T09:01:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_11.status 2024-09-11T09:01:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.status 2024-09-11T09:01:41.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_11.status 2024-09-11T09:01:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_11.status 2024-09-11T09:01:41.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.status 2024-09-11T09:01:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_11 2024-09-11T09:01:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:41.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_11 2024-09-11T09:01:41.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 106 2024-09-11T09:01:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_11 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.status 2024-09-11T09:01:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:41.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.status 2024-09-11T09:01:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_11 --format xml 2024-09-11T09:01:41.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:41.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=106 2024-09-11T09:01:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:41.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:42.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.status 2024-09-11T09:01:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.status 2024-09-11T09:01:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-11T09:01:42.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:42.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:42.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 106 = '' 2024-09-11T09:01:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:42.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.status 2024-09-11T09:01:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:42.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.status 2024-09-11T09:01:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-11T09:01:42.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:42.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:42.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:42.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 106 = '' 2024-09-11T09:01:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:42.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_11 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.status 2024-09-11T09:01:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.status 2024-09-11T09:01:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_11 --format xml 2024-09-11T09:01:43.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:43.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=106 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 106 = 106 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_11 snap 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:43.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:43.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:44.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_11@snap 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_11': 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 128c1a4db836 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.128c1a4db836 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:44.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:52:53 2024 2024-09-11T09:01:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:52:53 2024 2024-09-11T09:01:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:52:53 2024 2024-09-11T09:01:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: e101a667-f9d8-4d05-a046-469b05b5b3df 2024-09-11T09:01:44.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_11 snap 2024-09-11T09:01:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:01:44.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.export 2024-09-11T09:01:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.export 2024-09-11T09:01:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.export 2024-09-11T09:01:44.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_11@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.export 2024-09-11T09:01:44.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:44.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_11@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.export 2024-09-11T09:01:45.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:45.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.export 2024-09-11T09:01:45.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_11.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_11.export 2024-09-11T09:01:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_12 2024-09-11T09:01:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_12 snap 2024-09-11T09:01:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:45.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:45.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_12@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_12 2024-09-11T09:01:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:46.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_12 started 2024-09-11T09:01:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:46.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:47.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_12 started 2024-09-11T09:01:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:47.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_12 2024-09-11T09:01:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:47.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:47.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:47.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_12 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:47.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:47.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:47.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:47.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:47.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:47.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_12 2024-09-11T09:01:48.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:48.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:48.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_12 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:48.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_12 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_12.status 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_12.status 2024-09-11T09:01:48.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.status 2024-09-11T09:01:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_12.status 2024-09-11T09:01:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_12.status 2024-09-11T09:01:48.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_12 2024-09-11T09:01:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:48.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:48.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_12 2024-09-11T09:01:48.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 108 2024-09-11T09:01:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_12 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.status 2024-09-11T09:01:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:48.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.status 2024-09-11T09:01:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_12 --format xml 2024-09-11T09:01:48.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:48.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=108 2024-09-11T09:01:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:48.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:49.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T09:01:49.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:49.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=46 2024-09-11T09:01:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 108 = 46 2024-09-11T09:01:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:49.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:49.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T09:01:49.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:49.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=46 2024-09-11T09:01:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 108 = 46 2024-09-11T09:01:49.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:49.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:50.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T09:01:50.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:50.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:50.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 108 = '' 2024-09-11T09:01:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:50.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:01:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_12 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:52.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.status 2024-09-11T09:01:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_12 --format xml 2024-09-11T09:01:52.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:52.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=108 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 108 = 108 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_12 snap 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:52.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:52.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:53.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_12@snap 2024-09-11T09:01:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_12': 2024-09-11T09:01:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:01:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:01:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:01:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 128c4ab54931 2024-09-11T09:01:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.128c4ab54931 2024-09-11T09:01:53.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:06 2024 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:06 2024 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:06 2024 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 0791c4fe-f9ee-44f1-8b14-208766525577 2024-09-11T09:01:53.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:01:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:53.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_12 snap 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.export 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.export 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.export 2024-09-11T09:01:53.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_12@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.export 2024-09-11T09:01:53.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:53.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_12@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.export 2024-09-11T09:01:53.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:01:53.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.export 2024-09-11T09:01:54.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_12.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_12.export 2024-09-11T09:01:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:01:54.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_13 2024-09-11T09:01:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_13 snap 2024-09-11T09:01:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:54.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_13@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:01:55.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_13 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_13 started 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:55.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:01:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_13 started 2024-09-11T09:01:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:01:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:01:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:01:56.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_13 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:01:56.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:01:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:01:56.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:01:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:01:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_13 2024-09-11T09:01:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:01:56.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:01:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:01:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:01:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:01:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:01:56.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:01:56.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:01:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:01:56.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_13 2024-09-11T09:01:57.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:01:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:01:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:01:57.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:01:57.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:01:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:01:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:01:57.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_13 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:01:57.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_13 2024-09-11T09:01:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:01:57.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_13.status 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_13.status 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.status 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_13.status 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_13.status 2024-09-11T09:01:57.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:01:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.status 2024-09-11T09:01:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_13 2024-09-11T09:01:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:57.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_13 2024-09-11T09:01:57.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 110 2024-09-11T09:01:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_13 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.status 2024-09-11T09:01:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:01:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.status 2024-09-11T09:01:57.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_13 --format xml 2024-09-11T09:01:57.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:58.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:01:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=110 2024-09-11T09:01:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:58.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:58.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:01:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.status 2024-09-11T09:01:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.status 2024-09-11T09:01:58.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-11T09:01:58.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:58.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:58.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 110 = '' 2024-09-11T09:01:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:58.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:01:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.status 2024-09-11T09:01:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:58.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.status 2024-09-11T09:01:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-11T09:01:58.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:01:58.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:01:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:01:58.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:01:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:01:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 110 = '' 2024-09-11T09:01:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:01:58.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:01:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_13 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.status 2024-09-11T09:01:59.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.status 2024-09-11T09:01:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_13 --format xml 2024-09-11T09:01:59.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:01:59.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:01:59.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=110 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 110 = 110 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_13 snap 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:01:59.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:00.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_13@snap 2024-09-11T09:02:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_13': 2024-09-11T09:02:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ada633c4f4 2024-09-11T09:02:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ada633c4f4 2024-09-11T09:02:00.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:18 2024 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:18 2024 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:18 2024 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 551a4abb-edcd-4eb2-8d3f-d80000f55c42 2024-09-11T09:02:00.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_13 snap 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.export 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.export 2024-09-11T09:02:00.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.export 2024-09-11T09:02:00.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_13@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.export 2024-09-11T09:02:00.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:00.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_13@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.export 2024-09-11T09:02:01.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:01.264 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.export 2024-09-11T09:02:01.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_13.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_13.export 2024-09-11T09:02:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_14 2024-09-11T09:02:01.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_14 snap 2024-09-11T09:02:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:01.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_14@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_14 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_14 started 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:02.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:02.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_14 started 2024-09-11T09:02:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:03.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:03.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_14 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:03.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:03.589 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_14 2024-09-11T09:02:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:03.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:03.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:03.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:03.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:03.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_14 2024-09-11T09:02:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-11T09:02:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:03.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_14 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:02:03.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:02:03.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:02:03.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:02:03.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:02:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:03.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_14 2024-09-11T09:02:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-11T09:02:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:03.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:03.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_14 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:03.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:03.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:02:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:02:03.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:02:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:03.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_14 2024-09-11T09:02:04.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_14'] 2024-09-11T09:02:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:04.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_14 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:02:04.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:04.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:02:04.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:02:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:02:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:02:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:02:04.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_14 2024-09-11T09:02:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:04.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:04.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:04.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:04.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_14 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_14 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:04.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_14.status 2024-09-11T09:02:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_14.status 2024-09-11T09:02:04.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.status 2024-09-11T09:02:04.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_14.status 2024-09-11T09:02:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_14.status 2024-09-11T09:02:04.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.status 2024-09-11T09:02:04.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_14 2024-09-11T09:02:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:04.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_14 2024-09-11T09:02:04.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 112 2024-09-11T09:02:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_14 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.status 2024-09-11T09:02:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:04.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.status 2024-09-11T09:02:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:04.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_14 --format xml 2024-09-11T09:02:04.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=112 2024-09-11T09:02:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:04.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:02:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.status 2024-09-11T09:02:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:04.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.status 2024-09-11T09:02:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-11T09:02:04.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:04.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:04.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:04.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:04.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 112 = '' 2024-09-11T09:02:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:04.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.status 2024-09-11T09:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.status 2024-09-11T09:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-11T09:02:05.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:05.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:05.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 112 = '' 2024-09-11T09:02:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:05.419 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:02:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_14 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.status 2024-09-11T09:02:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:06.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.status 2024-09-11T09:02:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:06.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_14 --format xml 2024-09-11T09:02:06.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=112 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 112 = 112 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_14 snap 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:02:06.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:07.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_14@snap 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_14': 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10b3c882ab5b 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10b3c882ab5b 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:07.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:24 2024 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:24 2024 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:24 2024 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 4174ebf4-c3c8-4e27-a392-9c68a9fb6a29 2024-09-11T09:02:07.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_14 snap 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.export 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.export 2024-09-11T09:02:07.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.export 2024-09-11T09:02:07.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_14@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.export 2024-09-11T09:02:07.590 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:07.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_14@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.export 2024-09-11T09:02:07.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:07.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.export 2024-09-11T09:02:07.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_14.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_14.export 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_15 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_15 snap 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:08.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_15@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:09.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_15 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_15 started 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:09.615 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_15 started 2024-09-11T09:02:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:10.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_15 2024-09-11T09:02:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:10.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:10.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:10.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_15 2024-09-11T09:02:10.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:10.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:10.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:10.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:10.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:10.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_15 2024-09-11T09:02:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-11T09:02:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:10.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_15 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:02:10.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:02:10.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:10.766 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:02:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:02:10.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:02:10.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:10.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_15 2024-09-11T09:02:10.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_15'] 2024-09-11T09:02:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:10.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_15 2024-09-11T09:02:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:02:10.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:10.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:02:10.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:02:10.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:02:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:10.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_15 2024-09-11T09:02:11.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:11.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_15 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:11.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_15 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_15.status 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_15.status 2024-09-11T09:02:11.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:11.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.status 2024-09-11T09:02:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_15.status 2024-09-11T09:02:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_15.status 2024-09-11T09:02:11.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_15 2024-09-11T09:02:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:11.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_15 2024-09-11T09:02:11.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 114 2024-09-11T09:02:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_15 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.status 2024-09-11T09:02:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:11.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.status 2024-09-11T09:02:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_15 --format xml 2024-09-11T09:02:11.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:11.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=114 2024-09-11T09:02:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:11.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:02:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:11.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T09:02:11.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:11.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=49 2024-09-11T09:02:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 114 = 49 2024-09-11T09:02:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:11.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:02:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:12.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T09:02:12.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:12.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:12.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:12.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 114 = '' 2024-09-11T09:02:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:12.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:02:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T09:02:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:13.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:13.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:13.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 114 = '' 2024-09-11T09:02:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:13.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:02:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_15 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:14.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.status 2024-09-11T09:02:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_15 --format xml 2024-09-11T09:02:14.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:15.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:15.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=114 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 114 = 114 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_15 snap 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:15.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:02:15.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:16.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_15@snap 2024-09-11T09:02:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_15': 2024-09-11T09:02:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:16.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 118f987b629a 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.118f987b629a 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:31 2024 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:31 2024 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:31 2024 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 5f90212f-9abd-40ef-98b3-fab670ebcab5 2024-09-11T09:02:16.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:16.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_15 snap 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.export 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.export 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.export 2024-09-11T09:02:16.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_15@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.export 2024-09-11T09:02:16.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:16.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_15@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.export 2024-09-11T09:02:16.600 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:16.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.export 2024-09-11T09:02:16.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_15.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_15.export 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_16 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_16 snap 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:16.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_16@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:18.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_16 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_16 started 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:18.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:18.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:19.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_16 started 2024-09-11T09:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:19.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_16 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:19.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:19.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_16 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:19.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:19.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:19.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:19.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:19.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_16 2024-09-11T09:02:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_16'] 2024-09-11T09:02:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:19.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_16 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:19.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:19.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:02:19.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:02:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:19.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_16 2024-09-11T09:02:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:19.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:19.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:19.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_16 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_16 2024-09-11T09:02:19.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_16.status 2024-09-11T09:02:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_16.status 2024-09-11T09:02:19.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.status 2024-09-11T09:02:19.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_16.status 2024-09-11T09:02:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_16.status 2024-09-11T09:02:19.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_16 2024-09-11T09:02:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:19.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_16 2024-09-11T09:02:20.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 116 2024-09-11T09:02:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_16 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.status 2024-09-11T09:02:20.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.status 2024-09-11T09:02:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_16 --format xml 2024-09-11T09:02:20.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:20.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:20.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=116 2024-09-11T09:02:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:20.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:02:20.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T09:02:20.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:21.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=50 2024-09-11T09:02:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 116 = 50 2024-09-11T09:02:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:21.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:02:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:21.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T09:02:21.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:21.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:21.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:21.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 116 = '' 2024-09-11T09:02:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:21.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:02:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:22.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T09:02:22.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:22.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:22.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:22.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 116 = '' 2024-09-11T09:02:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:22.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:02:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_16 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:23.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.status 2024-09-11T09:02:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_16 --format xml 2024-09-11T09:02:23.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:24.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:24.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=116 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 116 = 116 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_16 snap 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:02:24.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:25.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_16@snap 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_16': 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 128c7656576b 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.128c7656576b 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:25.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:46 2024 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:46 2024 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:46 2024 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 82fa71ee-6b32-459f-a37e-db8386916102 2024-09-11T09:02:25.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_16 snap 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.export 2024-09-11T09:02:25.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.export 2024-09-11T09:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.export 2024-09-11T09:02:25.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_16@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.export 2024-09-11T09:02:25.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:25.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_16@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.export 2024-09-11T09:02:25.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:25.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.export 2024-09-11T09:02:25.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_16.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_16.export 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_17 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_17 snap 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:25.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_17@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:27.293 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T09:02:27.293 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T09:02:27.293 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T09:02:27.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:02:27.294+0000 7f2defb4d640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 41353) UID: 1000 2024-09-11T09:02:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_17 2024-09-11T09:02:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_17 started 2024-09-11T09:02:27.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:27.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_17 started 2024-09-11T09:02:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:28.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:28.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_17 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:28.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:28.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:28.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_17 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:28.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:28.683 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:28.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:28.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:28.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:28.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_17 2024-09-11T09:02:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:28.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:28.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:28.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:28.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_17 2024-09-11T09:02:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:28.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_17 2024-09-11T09:02:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_17.status 2024-09-11T09:02:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_17.status 2024-09-11T09:02:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.status 2024-09-11T09:02:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_17.status 2024-09-11T09:02:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_17.status 2024-09-11T09:02:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_17 2024-09-11T09:02:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:28.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_17 2024-09-11T09:02:29.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 118 2024-09-11T09:02:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_17 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.status 2024-09-11T09:02:29.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.status 2024-09-11T09:02:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_17 --format xml 2024-09-11T09:02:29.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:29.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=118 2024-09-11T09:02:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:29.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:02:29.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T09:02:29.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:30.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:30.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:30.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 118 = '' 2024-09-11T09:02:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:30.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:02:30.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T09:02:30.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:30.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:30.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 118 = '' 2024-09-11T09:02:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:02:31.294 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T09:02:31.294 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:02:31.295 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T09:02:31.295 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T09:02:31.296 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T09:02:31.299 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T09:02:31.299 INFO:tasks.rbd_mirror_thrash:waiting for 96 secs before thrashing 2024-09-11T09:02:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T09:02:31.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:31.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:31.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:31.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 118 = '' 2024-09-11T09:02:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:31.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:02:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_17 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:32.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.status 2024-09-11T09:02:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_17 --format xml 2024-09-11T09:02:32.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=118 2024-09-11T09:02:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 118 = 118 2024-09-11T09:02:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_17 snap 2024-09-11T09:02:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:33.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:02:33.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:34.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_17@snap 2024-09-11T09:02:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_17': 2024-09-11T09:02:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:34.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ad53550994 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ad53550994 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:53:54 2024 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:53:54 2024 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:53:54 2024 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 28968644-b7f0-40d0-b9ce-2784d7675298 2024-09-11T09:02:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_17 snap 2024-09-11T09:02:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:02:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:34.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.export 2024-09-11T09:02:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.export 2024-09-11T09:02:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.export 2024-09-11T09:02:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_17@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.export 2024-09-11T09:02:34.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:34.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_17@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.export 2024-09-11T09:02:34.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:34.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.export 2024-09-11T09:02:34.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_17.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_17.export 2024-09-11T09:02:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_18 2024-09-11T09:02:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_18 snap 2024-09-11T09:02:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:34.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:34.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_18@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_18 2024-09-11T09:02:36.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_18 started 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:36.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_18 started 2024-09-11T09:02:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:37.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:37.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_18 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:37.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:37.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:37.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_18 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:37.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:37.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:37.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:37.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_18 2024-09-11T09:02:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-11T09:02:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_18 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:02:37.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:02:37.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:02:37.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:02:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:02:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:37.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:37.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_18 2024-09-11T09:02:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-11T09:02:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:38.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:38.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_18 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:38.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:02:38.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:38.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:02:38.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:02:38.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:38.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_18 2024-09-11T09:02:38.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_18'] 2024-09-11T09:02:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:38.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_18 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:02:38.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:02:38.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:02:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:02:38.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:38.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:02:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:02:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:02:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:02:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:02:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:02:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_18 2024-09-11T09:02:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:38.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:38.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:38.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:38.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_18 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_18 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:38.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_18.status 2024-09-11T09:02:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_18.status 2024-09-11T09:02:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:38.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.status 2024-09-11T09:02:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_18.status 2024-09-11T09:02:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_18.status 2024-09-11T09:02:38.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.status 2024-09-11T09:02:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_18 2024-09-11T09:02:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:38.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:38.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_18 2024-09-11T09:02:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 120 2024-09-11T09:02:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_18 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.status 2024-09-11T09:02:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:38.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.status 2024-09-11T09:02:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_18 --format xml 2024-09-11T09:02:38.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:38.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=120 2024-09-11T09:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:38.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:02:38.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.status 2024-09-11T09:02:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.status 2024-09-11T09:02:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-11T09:02:39.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:39.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:39.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 120 = '' 2024-09-11T09:02:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:02:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.status 2024-09-11T09:02:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:39.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.status 2024-09-11T09:02:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-11T09:02:39.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:39.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:39.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:39.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 120 = '' 2024-09-11T09:02:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:39.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:02:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_18 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.status 2024-09-11T09:02:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:40.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.status 2024-09-11T09:02:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_18 --format xml 2024-09-11T09:02:40.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:40.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=120 2024-09-11T09:02:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 120 = 120 2024-09-11T09:02:40.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_18 snap 2024-09-11T09:02:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:40.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:02:40.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:41.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_18@snap 2024-09-11T09:02:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_18': 2024-09-11T09:02:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:41.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10b32f042540 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10b32f042540 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:00 2024 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:00 2024 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:00 2024 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 350bf260-3a2c-48eb-8813-d7dafe84bdb1 2024-09-11T09:02:41.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:41.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_18 snap 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.export 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.export 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.export 2024-09-11T09:02:41.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_18@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.export 2024-09-11T09:02:41.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:41.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_18@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.export 2024-09-11T09:02:42.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:42.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.export 2024-09-11T09:02:42.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_18.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_18.export 2024-09-11T09:02:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_19 2024-09-11T09:02:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_19 snap 2024-09-11T09:02:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:42.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:42.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_19@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_19 2024-09-11T09:02:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:43.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_19 started 2024-09-11T09:02:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:43.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_19 started 2024-09-11T09:02:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:44.736 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_19 2024-09-11T09:02:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:44.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:44.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:44.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_19 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:44.742 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:44.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:44.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:44.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:44.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_19 2024-09-11T09:02:44.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:44.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:44.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:44.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_19 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:44.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_19 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_19.status 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_19.status 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:44.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.status 2024-09-11T09:02:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_19.status 2024-09-11T09:02:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_19.status 2024-09-11T09:02:44.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.status 2024-09-11T09:02:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_19 2024-09-11T09:02:44.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:44.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_19 2024-09-11T09:02:45.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 122 2024-09-11T09:02:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_19 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.status 2024-09-11T09:02:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:45.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.status 2024-09-11T09:02:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_19 --format xml 2024-09-11T09:02:45.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:45.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:45.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=122 2024-09-11T09:02:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:45.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:02:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.status 2024-09-11T09:02:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:46.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.status 2024-09-11T09:02:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-11T09:02:46.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:46.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:46.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 122 = '' 2024-09-11T09:02:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:46.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:02:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.status 2024-09-11T09:02:46.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.status 2024-09-11T09:02:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-11T09:02:46.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:46.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:46.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:46.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 122 = '' 2024-09-11T09:02:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:46.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:02:47.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_19 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.status 2024-09-11T09:02:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.status 2024-09-11T09:02:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_19 --format xml 2024-09-11T09:02:47.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:47.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=122 2024-09-11T09:02:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 122 = 122 2024-09-11T09:02:47.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_19 snap 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:02:47.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:48.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_19@snap 2024-09-11T09:02:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_19': 2024-09-11T09:02:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 118f4ce3ce7f 2024-09-11T09:02:48.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.118f4ce3ce7f 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:07 2024 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:07 2024 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:07 2024 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 05c8607a-0bf6-407e-b458-e04320b3a7ac 2024-09-11T09:02:48.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_19 snap 2024-09-11T09:02:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:02:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:48.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.export 2024-09-11T09:02:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.export 2024-09-11T09:02:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.export 2024-09-11T09:02:48.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_19@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.export 2024-09-11T09:02:48.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:48.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_19@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.export 2024-09-11T09:02:48.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:49.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.export 2024-09-11T09:02:49.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_19.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_19.export 2024-09-11T09:02:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_20 2024-09-11T09:02:49.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_20 snap 2024-09-11T09:02:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:49.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_20@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_20 2024-09-11T09:02:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_20 started 2024-09-11T09:02:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:50.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:50.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:51.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_20 started 2024-09-11T09:02:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:51.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_20 2024-09-11T09:02:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:51.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:51.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:51.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_20 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:51.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:51.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:51.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:51.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_20 2024-09-11T09:02:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_20'] 2024-09-11T09:02:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:51.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_20 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:02:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:02:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:02:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:02:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:02:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:02:51.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:02:51.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:02:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_20 2024-09-11T09:02:52.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:52.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:52.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_20 2024-09-11T09:02:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_20 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:52.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_20.status 2024-09-11T09:02:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_20.status 2024-09-11T09:02:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.status 2024-09-11T09:02:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_20.status 2024-09-11T09:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_20.status 2024-09-11T09:02:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.status 2024-09-11T09:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_20 2024-09-11T09:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_20 2024-09-11T09:02:52.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 124 2024-09-11T09:02:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_20 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.status 2024-09-11T09:02:52.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.status 2024-09-11T09:02:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_20 --format xml 2024-09-11T09:02:52.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:52.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=124 2024-09-11T09:02:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:52.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:02:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.status 2024-09-11T09:02:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:53.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.status 2024-09-11T09:02:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-11T09:02:53.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:53.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:53.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 124 = '' 2024-09-11T09:02:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:53.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:02:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.status 2024-09-11T09:02:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:53.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.status 2024-09-11T09:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-11T09:02:53.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:53.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:53.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:02:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:02:53.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:02:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:02:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 124 = '' 2024-09-11T09:02:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:53.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:02:54.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_20 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.status 2024-09-11T09:02:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.status 2024-09-11T09:02:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_20 --format xml 2024-09-11T09:02:54.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:54.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:02:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=124 2024-09-11T09:02:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 124 = 124 2024-09-11T09:02:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:54.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_20 snap 2024-09-11T09:02:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:02:54.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:55.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_20@snap 2024-09-11T09:02:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_20': 2024-09-11T09:02:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:02:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:02:55.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 128cb0ceaffa 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.128cb0ceaffa 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:21 2024 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:21 2024 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:21 2024 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:02:55.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:02:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:02:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 6bae77b4-74ad-4d05-87e9-78bd958ad83a 2024-09-11T09:02:55.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:02:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_20 snap 2024-09-11T09:02:55.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:02:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:02:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.export 2024-09-11T09:02:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.export 2024-09-11T09:02:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.export 2024-09-11T09:02:55.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_20@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.export 2024-09-11T09:02:55.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:55.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_20@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.export 2024-09-11T09:02:56.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:02:56.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.export 2024-09-11T09:02:56.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_20.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_20.export 2024-09-11T09:02:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:56.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:02:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_21 2024-09-11T09:02:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_21 snap 2024-09-11T09:02:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:02:56.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_21@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_21 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_21 started 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:57.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:02:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:02:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:57.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:02:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_21 started 2024-09-11T09:02:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:02:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:02:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:02:58.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:02:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_21 2024-09-11T09:02:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:02:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:02:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:02:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:02:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:02:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:02:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:02:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:02:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_21 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:58.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:02:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:02:58.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:58.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:02:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:02:58.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:02:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:02:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_21 2024-09-11T09:02:58.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-11T09:02:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:58.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:58.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_21 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:58.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:02:58.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:02:58.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:02:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:02:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:02:58.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:02:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:02:58.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_21 2024-09-11T09:02:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_21'] 2024-09-11T09:02:59.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:02:59.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:02:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:02:59.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_21 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:59.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:02:59.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:02:59.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:02:59.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:02:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:02:59.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_21 2024-09-11T09:02:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:02:59.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:02:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:02:59.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:02:59.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:02:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:02:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:02:59.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_21 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_21 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:02:59.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_21.status 2024-09-11T09:02:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_21.status 2024-09-11T09:02:59.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.status 2024-09-11T09:02:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_21.status 2024-09-11T09:02:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_21.status 2024-09-11T09:02:59.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:02:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:02:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_21 2024-09-11T09:02:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:59.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:59.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_21 2024-09-11T09:02:59.398 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8867304415445795) > weight (0.25) 2024-09-11T09:02:59.398 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-11T09:02:59.399 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-11T09:02:59.399 INFO:tasks.rbd_mirror_thrash:waiting for 8 secs before reviving daemons 2024-09-11T09:02:59.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:02:59.397+0000 7f73968cc640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 30797) UID: 1000 2024-09-11T09:02:59.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 126 2024-09-11T09:02:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_21 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.status 2024-09-11T09:02:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:02:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:02:59.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:02:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.status 2024-09-11T09:02:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_21 --format xml 2024-09-11T09:02:59.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:02:59.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:02:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=126 2024-09-11T09:02:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:02:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:02:59.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:00.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T09:03:00.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:00.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:00.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 126 = '' 2024-09-11T09:03:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:00.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:03:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:00.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T09:03:00.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:00.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:00.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 126 = '' 2024-09-11T09:03:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:00.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:03:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:01.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T09:03:01.469 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:01.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:01.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:01.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 126 = '' 2024-09-11T09:03:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:01.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:03:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:03.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T09:03:03.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:03.213 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=55 2024-09-11T09:03:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 126 = 55 2024-09-11T09:03:03.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:03.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 2 2024-09-11T09:03:05.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:05.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T09:03:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:05.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:05.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 126 = '' 2024-09-11T09:03:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:05.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 2 2024-09-11T09:03:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_21 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:07.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.status 2024-09-11T09:03:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_21 --format xml 2024-09-11T09:03:07.333 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:07.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:07.400 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-11T09:03:07.400 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:03:07.400 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-11T09:03:07.400 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-11T09:03:07.400 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=126 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 126 = 126 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_21 snap 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:07.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:03:07.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:07.404 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-11T09:03:07.404 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-11T09:03:08.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_21@snap 2024-09-11T09:03:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_21': 2024-09-11T09:03:08.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1294ff255d03 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1294ff255d03 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:31 2024 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:31 2024 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:31 2024 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:03:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:03:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: ae0c9716-d47a-4a34-8cf9-e01df79ded5c 2024-09-11T09:03:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:03:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_21 snap 2024-09-11T09:03:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:03:08.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.export 2024-09-11T09:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.export 2024-09-11T09:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.export 2024-09-11T09:03:08.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_21@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.export 2024-09-11T09:03:08.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:08.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_21@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.export 2024-09-11T09:03:09.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:09.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.export 2024-09-11T09:03:09.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_21.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_21.export 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_22 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_22 snap 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:09.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_22@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:03:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_22 2024-09-11T09:03:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_22 started 2024-09-11T09:03:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:10.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:03:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_22 started 2024-09-11T09:03:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:03:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:03:11.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:03:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_22 2024-09-11T09:03:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:03:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:03:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:03:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:03:11.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:03:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:03:11.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:11.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:11.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:11.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:11.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:11.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-11T09:03:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-11T09:03:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:11.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:11.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:03:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:03:11.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:03:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:03:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:11.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:03:11.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:03:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:12.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-11T09:03:12.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-11T09:03:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:12.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_22 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:12.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:03:12.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:12.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:03:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:03:12.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:03:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:12.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_22 2024-09-11T09:03:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-11T09:03:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:12.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_22 2024-09-11T09:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:12.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:03:12.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:03:12.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:03:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:03:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:03:12.258 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:03:12.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:03:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:03:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:03:12.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_22 2024-09-11T09:03:12.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-11T09:03:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:12.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:12.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 1 2024-09-11T09:03:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:13.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_22 2024-09-11T09:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:13.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:13.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:13.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:13.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:13.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_22 2024-09-11T09:03:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_22'] 2024-09-11T09:03:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:13.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_22 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:13.550 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:03:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:03:13.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:13.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:03:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:03:13.554 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:03:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:13.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:13.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_22 2024-09-11T09:03:13.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:03:13.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:03:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:03:13.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:03:13.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:03:13.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_22 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_22 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:13.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_22.status 2024-09-11T09:03:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_22.status 2024-09-11T09:03:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:13.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.status 2024-09-11T09:03:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_22.status 2024-09-11T09:03:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_22.status 2024-09-11T09:03:13.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.status 2024-09-11T09:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_22 2024-09-11T09:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:13.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_22 2024-09-11T09:03:13.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 128 2024-09-11T09:03:13.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_22 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.status 2024-09-11T09:03:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.status 2024-09-11T09:03:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_22 --format xml 2024-09-11T09:03:13.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:13.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:03:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=128 2024-09-11T09:03:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:13.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.status 2024-09-11T09:03:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:14.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.status 2024-09-11T09:03:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-11T09:03:14.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:14.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:14.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 128 = '' 2024-09-11T09:03:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:14.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:03:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.status 2024-09-11T09:03:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:14.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.status 2024-09-11T09:03:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-11T09:03:14.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:14.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:14.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 128 = '' 2024-09-11T09:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:14.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:03:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_22 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.status 2024-09-11T09:03:15.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.status 2024-09-11T09:03:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_22 --format xml 2024-09-11T09:03:15.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:15.499 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:15.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=128 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 128 = 128 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_22 snap 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:03:15.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:16.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_22@snap 2024-09-11T09:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_22': 2024-09-11T09:03:16.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12942aa615aa 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12942aa615aa 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:38 2024 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:38 2024 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:38 2024 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:03:16.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:03:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 9ceefcdb-b3e9-4601-8f51-a204c212530d 2024-09-11T09:03:16.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:03:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_22 snap 2024-09-11T09:03:16.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:03:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:03:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.export 2024-09-11T09:03:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.export 2024-09-11T09:03:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.export 2024-09-11T09:03:16.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_22@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.export 2024-09-11T09:03:17.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:17.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_22@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.export 2024-09-11T09:03:17.407 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:17.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.export 2024-09-11T09:03:17.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_22.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_22.export 2024-09-11T09:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_23 2024-09-11T09:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_23 snap 2024-09-11T09:03:17.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:17.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_23@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_23 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_23 started 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:18.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:18.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:19.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_23 started 2024-09-11T09:03:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:03:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:03:19.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:03:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_23 2024-09-11T09:03:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:03:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:03:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:03:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:03:19.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:03:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:03:19.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:19.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_23 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:19.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:19.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:19.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:19.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:19.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_23 2024-09-11T09:03:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:03:20.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:03:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:03:20.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:03:20.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_23 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_23 2024-09-11T09:03:20.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_23.status 2024-09-11T09:03:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:20.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_23.status 2024-09-11T09:03:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.status 2024-09-11T09:03:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_23.status 2024-09-11T09:03:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_23.status 2024-09-11T09:03:20.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.status 2024-09-11T09:03:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_23 2024-09-11T09:03:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:20.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_23 2024-09-11T09:03:20.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 130 2024-09-11T09:03:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_23 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.status 2024-09-11T09:03:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:20.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.status 2024-09-11T09:03:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_23 --format xml 2024-09-11T09:03:20.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:20.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:03:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=130 2024-09-11T09:03:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:20.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:20.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.status 2024-09-11T09:03:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:21.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.status 2024-09-11T09:03:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-11T09:03:21.161 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:21.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:21.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 130 = '' 2024-09-11T09:03:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:21.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:03:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.status 2024-09-11T09:03:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:21.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.status 2024-09-11T09:03:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-11T09:03:21.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:21.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:21.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 130 = '' 2024-09-11T09:03:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:21.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:03:22.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_23 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.status 2024-09-11T09:03:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.status 2024-09-11T09:03:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_23 --format xml 2024-09-11T09:03:22.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:22.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=130 2024-09-11T09:03:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 130 = 130 2024-09-11T09:03:22.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_23 snap 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:03:22.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:23.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_23@snap 2024-09-11T09:03:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_23': 2024-09-11T09:03:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:03:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:03:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:03:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 128c2e0a5fad 2024-09-11T09:03:23.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.128c2e0a5fad 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:49 2024 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:49 2024 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:49 2024 2024-09-11T09:03:23.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:03:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:03:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:03:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 69e0ab8a-356e-4e12-8363-b402fa05e07f 2024-09-11T09:03:23.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:03:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:23.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_23 snap 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.export 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.export 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.export 2024-09-11T09:03:23.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_23@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.export 2024-09-11T09:03:23.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:23.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_23@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.export 2024-09-11T09:03:24.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:24.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.export 2024-09-11T09:03:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_23.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_23.export 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_24 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_24 snap 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:24.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_24@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:03:25.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_24 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_24 started 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:25.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_24 started 2024-09-11T09:03:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:03:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:03:26.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:03:26.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_24 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:26.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:03:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:26.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_24 2024-09-11T09:03:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:26.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:26.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:26.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:26.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:26.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:26.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_24 2024-09-11T09:03:27.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:03:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:03:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:03:27.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:03:27.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:03:27.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_24 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_24 2024-09-11T09:03:27.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_24.status 2024-09-11T09:03:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_24.status 2024-09-11T09:03:27.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.status 2024-09-11T09:03:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_24.status 2024-09-11T09:03:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_24.status 2024-09-11T09:03:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.status 2024-09-11T09:03:27.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_24 2024-09-11T09:03:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:27.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_24 2024-09-11T09:03:27.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 132 2024-09-11T09:03:27.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_24 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.status 2024-09-11T09:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.status 2024-09-11T09:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_24 --format xml 2024-09-11T09:03:27.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:27.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:03:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=132 2024-09-11T09:03:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:27.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.status 2024-09-11T09:03:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:28.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.status 2024-09-11T09:03:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-11T09:03:28.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:28.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:28.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 132 = '' 2024-09-11T09:03:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:28.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:03:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.status 2024-09-11T09:03:28.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.status 2024-09-11T09:03:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-11T09:03:28.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:28.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:28.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:28.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 132 = '' 2024-09-11T09:03:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:28.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:03:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_24 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.status 2024-09-11T09:03:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:29.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.status 2024-09-11T09:03:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_24 --format xml 2024-09-11T09:03:29.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:29.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=132 2024-09-11T09:03:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 132 = 132 2024-09-11T09:03:29.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_24 snap 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:03:29.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:30.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_24@snap 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_24': 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12944bf2d759 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12944bf2d759 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:03:30.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:54:59 2024 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:54:59 2024 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:54:59 2024 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 523e3106-9d30-48a3-8d7d-0a45f9581e5e 2024-09-11T09:03:30.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:03:30.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_24 snap 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.export 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.export 2024-09-11T09:03:30.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.export 2024-09-11T09:03:30.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_24@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.export 2024-09-11T09:03:30.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:30.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_24@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.export 2024-09-11T09:03:31.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:31.234 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.export 2024-09-11T09:03:31.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_24.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_24.export 2024-09-11T09:03:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:03:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_25 2024-09-11T09:03:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_25 snap 2024-09-11T09:03:31.351 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:31.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_25@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:03:32.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_25 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_25 started 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:32.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:03:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:32.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:33.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_25 started 2024-09-11T09:03:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:03:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:03:33.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:03:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_25 2024-09-11T09:03:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:03:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:03:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:03:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:03:33.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:03:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:03:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:33.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:33.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_25 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:33.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:33.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:33.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:33.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:33.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_25 2024-09-11T09:03:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:03:34.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:03:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:03:34.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:03:34.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_25 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:03:34.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_25 2024-09-11T09:03:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_25.status 2024-09-11T09:03:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_25.status 2024-09-11T09:03:34.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.status 2024-09-11T09:03:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_25.status 2024-09-11T09:03:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_25.status 2024-09-11T09:03:34.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_25 2024-09-11T09:03:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:34.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_25 2024-09-11T09:03:34.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 134 2024-09-11T09:03:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_25 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.status 2024-09-11T09:03:34.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.status 2024-09-11T09:03:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:34.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_25 --format xml 2024-09-11T09:03:35.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:03:35.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=134 2024-09-11T09:03:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:35.213 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T09:03:35.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=59 2024-09-11T09:03:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 134 = 59 2024-09-11T09:03:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:35.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:35.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:35.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T09:03:35.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=59 2024-09-11T09:03:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 134 = 59 2024-09-11T09:03:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:35.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:03:36.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T09:03:36.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:36.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:36.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 134 = '' 2024-09-11T09:03:36.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:36.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:03:38.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_25 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.status 2024-09-11T09:03:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_25 --format xml 2024-09-11T09:03:38.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:38.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=134 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 134 = 134 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_25 snap 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:38.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:03:38.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:39.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_25@snap 2024-09-11T09:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_25': 2024-09-11T09:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ad685dcb1b 2024-09-11T09:03:39.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ad685dcb1b 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:09 2024 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:09 2024 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:09 2024 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 0534cedd-e737-4e3d-87e8-aff6ea511c46 2024-09-11T09:03:39.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:03:39.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_25 snap 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.export 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.export 2024-09-11T09:03:39.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.export 2024-09-11T09:03:39.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_25@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.export 2024-09-11T09:03:39.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_25@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.export 2024-09-11T09:03:39.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:39.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.export 2024-09-11T09:03:39.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_25.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_25.export 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_26 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_26 snap 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:39.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_26@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_26 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_26 started 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:03:41.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:41.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_26 started 2024-09-11T09:03:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:42.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:03:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:03:42.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_26 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:42.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:03:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:42.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_26 2024-09-11T09:03:42.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:42.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:42.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:42.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_26 2024-09-11T09:03:43.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-11T09:03:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:43.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_26 2024-09-11T09:03:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:03:43.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:03:43.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:43.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:03:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:03:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:03:43.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:03:43.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:03:43.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:43.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_26 2024-09-11T09:03:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_26'] 2024-09-11T09:03:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:43.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_26 2024-09-11T09:03:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:03:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:43.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:03:43.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:03:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:03:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:03:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:43.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:03:43.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:03:43.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:43.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_26 2024-09-11T09:03:43.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:03:43.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:03:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:03:43.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:03:43.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:03:43.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_26 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_26 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:43.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_26.status 2024-09-11T09:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_26.status 2024-09-11T09:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:43.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.status 2024-09-11T09:03:43.400 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_26.status 2024-09-11T09:03:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_26.status 2024-09-11T09:03:43.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_26 2024-09-11T09:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:43.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_26 2024-09-11T09:03:43.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 136 2024-09-11T09:03:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_26 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.status 2024-09-11T09:03:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:43.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.status 2024-09-11T09:03:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_26 --format xml 2024-09-11T09:03:43.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:44.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:03:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=136 2024-09-11T09:03:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:44.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:44.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T09:03:44.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:44.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=60 2024-09-11T09:03:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 136 = 60 2024-09-11T09:03:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:44.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:03:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:44.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T09:03:44.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:44.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=60 2024-09-11T09:03:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 136 = 60 2024-09-11T09:03:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:44.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:03:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:45.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T09:03:45.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:45.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:45.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:45.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 136 = '' 2024-09-11T09:03:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:45.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:03:47.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_26 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.status 2024-09-11T09:03:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_26 --format xml 2024-09-11T09:03:47.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:47.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:47.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=136 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 136 = 136 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_26 snap 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:03:47.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:48.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_26@snap 2024-09-11T09:03:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_26': 2024-09-11T09:03:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:03:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:03:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:03:48.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10b31489ed84 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10b31489ed84 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:16 2024 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:16 2024 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:16 2024 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: de26d284-def3-43fc-8308-3c61d65c9c5f 2024-09-11T09:03:48.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:03:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_26 snap 2024-09-11T09:03:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:03:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:48.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:03:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.export 2024-09-11T09:03:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.export 2024-09-11T09:03:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.export 2024-09-11T09:03:48.393 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_26@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.export 2024-09-11T09:03:48.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:48.612 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_26@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.export 2024-09-11T09:03:48.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:48.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.export 2024-09-11T09:03:49.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_26.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_26.export 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_27 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_27 snap 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:49.107 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_27@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:03:50.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_27 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_27 started 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:50.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:50.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_27 started 2024-09-11T09:03:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:03:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:03:51.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:03:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_27 2024-09-11T09:03:51.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:03:51.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:51.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_27 2024-09-11T09:03:51.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:51.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:51.650 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:51.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:51.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:51.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:51.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_27 2024-09-11T09:03:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-11T09:03:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:51.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_27 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:51.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:03:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:03:51.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:03:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:03:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:51.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:03:51.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:03:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:51.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_27 2024-09-11T09:03:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_27'] 2024-09-11T09:03:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:51.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_27 2024-09-11T09:03:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:03:51.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:03:51.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:03:51.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:03:51.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:03:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:51.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_27 2024-09-11T09:03:52.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:03:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:03:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:03:52.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:03:52.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:03:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:03:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:03:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_27 2024-09-11T09:03:52.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_27 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_27.status 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_27.status 2024-09-11T09:03:52.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:52.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.status 2024-09-11T09:03:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_27.status 2024-09-11T09:03:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_27.status 2024-09-11T09:03:52.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.status 2024-09-11T09:03:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_27 2024-09-11T09:03:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:52.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_27 2024-09-11T09:03:52.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 138 2024-09-11T09:03:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_27 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.status 2024-09-11T09:03:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:52.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.status 2024-09-11T09:03:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_27 --format xml 2024-09-11T09:03:52.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:52.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:03:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=138 2024-09-11T09:03:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:52.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:52.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.status 2024-09-11T09:03:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.status 2024-09-11T09:03:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-11T09:03:52.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:52.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=61 2024-09-11T09:03:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 138 = 61 2024-09-11T09:03:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:52.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:03:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.status 2024-09-11T09:03:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:53.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.status 2024-09-11T09:03:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-11T09:03:53.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:53.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:53.465 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:03:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:03:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:03:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:03:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 138 = '' 2024-09-11T09:03:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:53.466 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:03:54.268 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_27 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.status 2024-09-11T09:03:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.status 2024-09-11T09:03:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_27 --format xml 2024-09-11T09:03:54.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:54.333 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=138 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 138 = 138 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_27 snap 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:54.336 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:03:54.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:55.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_27@snap 2024-09-11T09:03:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_27': 2024-09-11T09:03:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:03:55.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 128c1bff14c6 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.128c1bff14c6 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:28 2024 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:28 2024 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:28 2024 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:03:55.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:03:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8c3ff342-3906-447a-9223-131a82f747f2 2024-09-11T09:03:55.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_27 snap 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.export 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.export 2024-09-11T09:03:55.402 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.export 2024-09-11T09:03:55.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_27@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.export 2024-09-11T09:03:55.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:55.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_27@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.export 2024-09-11T09:03:55.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:03:55.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.export 2024-09-11T09:03:55.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_27.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_27.export 2024-09-11T09:03:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:03:56.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_28 2024-09-11T09:03:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_28 snap 2024-09-11T09:03:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:03:56.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_28@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:03:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_28 2024-09-11T09:03:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:57.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_28 started 2024-09-11T09:03:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:03:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:03:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:57.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:03:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_28 started 2024-09-11T09:03:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:58.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:03:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:03:58.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_28 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:03:58.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:03:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:03:58.649 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:03:58.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_28 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:03:58.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:03:58.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:03:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:58.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:03:58.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:03:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:03:58.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_28 2024-09-11T09:03:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-11T09:03:58.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:58.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_28 2024-09-11T09:03:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:03:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:58.794 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:03:58.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:03:58.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:03:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:03:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_28 2024-09-11T09:03:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_28'] 2024-09-11T09:03:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:03:58.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_28 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:03:58.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:03:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:03:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:03:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:03:58.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:03:58.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:03:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:03:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:03:58.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:03:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:03:58.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_28 2024-09-11T09:03:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:03:59.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:03:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:03:59.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:03:59.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:03:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:03:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:03:59.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_28 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_28 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_28.status 2024-09-11T09:03:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_28.status 2024-09-11T09:03:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:59.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.status 2024-09-11T09:03:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_28.status 2024-09-11T09:03:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_28.status 2024-09-11T09:03:59.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:03:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:03:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_28 2024-09-11T09:03:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:59.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_28 2024-09-11T09:03:59.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 140 2024-09-11T09:03:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_28 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.status 2024-09-11T09:03:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:03:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:59.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.status 2024-09-11T09:03:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_28 --format xml 2024-09-11T09:03:59.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:03:59.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:03:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=140 2024-09-11T09:03:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:03:59.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:03:59.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:03:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:03:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:03:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:03:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:03:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:03:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T09:03:59.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:00.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:00.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=62 2024-09-11T09:04:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 140 = 62 2024-09-11T09:04:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:00.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:04:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:04:00.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:04:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T09:04:00.425 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:00.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:00.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 140 = '' 2024-09-11T09:04:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:00.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:04:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:04:01.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:04:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T09:04:01.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:01.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:01.365 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 140 = '' 2024-09-11T09:04:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:01.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:04:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_28 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:04:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:04:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.status 2024-09-11T09:04:02.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_28 --format xml 2024-09-11T09:04:02.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:03.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=140 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 140 = 140 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_28 snap 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:03.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:03.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:04.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_28@snap 2024-09-11T09:04:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_28': 2024-09-11T09:04:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:04.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12943827fd73 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12943827fd73 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:37 2024 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:37 2024 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:37 2024 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: fece848e-5f36-407c-83b5-17909bfc414d 2024-09-11T09:04:04.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_28 snap 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.export 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.export 2024-09-11T09:04:04.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.export 2024-09-11T09:04:04.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_28@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.export 2024-09-11T09:04:04.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:04.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_28@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.export 2024-09-11T09:04:04.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:04.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.export 2024-09-11T09:04:04.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_28.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_28.export 2024-09-11T09:04:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_29 2024-09-11T09:04:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_29 snap 2024-09-11T09:04:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:04.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:04.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_29@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_29 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_29 started 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:06.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:06.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:07.300 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.5762898539910893) > weight (0.25) 2024-09-11T09:04:07.300 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7608704061339747) > weight (0.25) 2024-09-11T09:04:07.301 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.4959528316581969) > weight (0.25) 2024-09-11T09:04:07.301 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.3 2024-09-11T09:04:07.301 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Sent signal 15 2024-09-11T09:04:07.301 INFO:tasks.rbd_mirror_thrash:waiting for 2 secs before reviving daemons 2024-09-11T09:04:07.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:04:07.304+0000 7fc1e6b40640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 30919) UID: 1000 2024-09-11T09:04:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_29 started 2024-09-11T09:04:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:07.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:07.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_29 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:07.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:07.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_29 2024-09-11T09:04:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:07.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:07.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:07.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:07.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:07.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_29 2024-09-11T09:04:07.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_29'] 2024-09-11T09:04:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:07.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_29 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:07.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:07.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:07.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:07.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:07.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_29 2024-09-11T09:04:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:08.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:08.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:08.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_29 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:08.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_29 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_29.status 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:08.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_29.status 2024-09-11T09:04:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.status 2024-09-11T09:04:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_29.status 2024-09-11T09:04:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_29.status 2024-09-11T09:04:08.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.status 2024-09-11T09:04:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_29 2024-09-11T09:04:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:08.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_29 2024-09-11T09:04:08.523 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:04:08.525+0000 7f6578d84640 -1 reset not still connected to 0x55b98ce4b040 2024-09-11T09:04:08.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 142 2024-09-11T09:04:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_29 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.status 2024-09-11T09:04:08.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.status 2024-09-11T09:04:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_29 --format xml 2024-09-11T09:04:08.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:08.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=142 2024-09-11T09:04:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:08.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:08.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.status 2024-09-11T09:04:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:08.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.status 2024-09-11T09:04:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-11T09:04:08.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:09.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:09.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:09.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 142 = '' 2024-09-11T09:04:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:09.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:09.303 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.3 2024-09-11T09:04:09.303 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:04:09.303 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.3 2024-09-11T09:04:09.303 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Restarting daemon 2024-09-11T09:04:09.304 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 2024-09-11T09:04:09.309 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Started 2024-09-11T09:04:09.310 INFO:tasks.rbd_mirror_thrash:waiting for 98 secs before thrashing 2024-09-11T09:04:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.status 2024-09-11T09:04:09.454 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.status 2024-09-11T09:04:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-11T09:04:09.455 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:09.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:09.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:09.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 142 = '' 2024-09-11T09:04:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:09.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_29 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.status 2024-09-11T09:04:10.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.status 2024-09-11T09:04:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_29 --format xml 2024-09-11T09:04:10.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:10.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:10.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=142 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 142 = 142 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_29 snap 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:10.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:11.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_29@snap 2024-09-11T09:04:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_29': 2024-09-11T09:04:11.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ad315d7fe 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ad315d7fe 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:55:47 2024 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:55:47 2024 2024-09-11T09:04:11.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:55:47 2024 2024-09-11T09:04:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 2a587c46-253d-496d-be6f-0d9d4063df36 2024-09-11T09:04:11.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:11.448 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_29 snap 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.export 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.export 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.export 2024-09-11T09:04:11.449 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_29@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.export 2024-09-11T09:04:11.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:11.665 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_29@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.export 2024-09-11T09:04:12.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:12.042 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.export 2024-09-11T09:04:12.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_29.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_29.export 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_30 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_30 snap 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:12.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_30@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_30 2024-09-11T09:04:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:13.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_30 started 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:13.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_30 started 2024-09-11T09:04:14.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:14.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:14.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_30 2024-09-11T09:04:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:14.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:14.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_30 2024-09-11T09:04:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:14.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:14.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:14.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:14.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:14.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:14.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:14.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:14.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_30 2024-09-11T09:04:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-11T09:04:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:14.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_30 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:14.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:14.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:14.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:14.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:14.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:14.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_30 2024-09-11T09:04:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-11T09:04:14.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:14.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:14.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_30 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:14.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:04:14.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:04:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:04:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:14.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:04:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:04:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:14.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:04:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:04:14.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:04:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:15.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_30 2024-09-11T09:04:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_30'] 2024-09-11T09:04:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:15.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_30 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:15.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:04:15.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:04:15.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:04:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:15.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:15.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_30 2024-09-11T09:04:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:15.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:15.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:15.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:15.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_30 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_30 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:15.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_30.status 2024-09-11T09:04:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_30.status 2024-09-11T09:04:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:15.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.status 2024-09-11T09:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_30.status 2024-09-11T09:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_30.status 2024-09-11T09:04:15.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.status 2024-09-11T09:04:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_30 2024-09-11T09:04:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:15.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_30 2024-09-11T09:04:15.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 144 2024-09-11T09:04:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_30 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.status 2024-09-11T09:04:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:15.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.status 2024-09-11T09:04:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_30 --format xml 2024-09-11T09:04:15.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:15.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=144 2024-09-11T09:04:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:15.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.status 2024-09-11T09:04:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:16.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.status 2024-09-11T09:04:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-11T09:04:16.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:16.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:16.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 144 = '' 2024-09-11T09:04:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:16.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:16.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.status 2024-09-11T09:04:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.status 2024-09-11T09:04:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-11T09:04:16.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:16.542 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:16.543 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:16.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 144 = '' 2024-09-11T09:04:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:16.546 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:17.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_30 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.status 2024-09-11T09:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.status 2024-09-11T09:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_30 --format xml 2024-09-11T09:04:17.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:17.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:17.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=144 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 144 = 144 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_30 snap 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:17.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:18.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_30@snap 2024-09-11T09:04:18.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_30': 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12a1c6166ad 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12a1c6166ad 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:24 2024 2024-09-11T09:04:18.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:24 2024 2024-09-11T09:04:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:24 2024 2024-09-11T09:04:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: eb74963f-2883-4117-8704-7dd16bb13e33 2024-09-11T09:04:18.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_30 snap 2024-09-11T09:04:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:04:18.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.export 2024-09-11T09:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.export 2024-09-11T09:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.export 2024-09-11T09:04:18.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_30@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.export 2024-09-11T09:04:18.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:18.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_30@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.export 2024-09-11T09:04:19.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:19.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.export 2024-09-11T09:04:19.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_30.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_30.export 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_31 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_31 snap 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:19.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_31@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_31 2024-09-11T09:04:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:20.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_31 started 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:20.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_31 started 2024-09-11T09:04:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:21.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_31 2024-09-11T09:04:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:21.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:21.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:21.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:21.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_31 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:21.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:21.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:21.762 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:21.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:21.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_31 2024-09-11T09:04:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-11T09:04:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:21.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_31 2024-09-11T09:04:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:21.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:21.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:21.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:21.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:21.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:21.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_31 2024-09-11T09:04:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-11T09:04:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:22.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_31 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:22.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:04:22.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:04:22.035 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:04:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:22.056 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_31 2024-09-11T09:04:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_31'] 2024-09-11T09:04:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:22.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:22.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_31 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:22.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:04:22.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:04:22.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:04:22.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:04:22.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:22.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_31 2024-09-11T09:04:22.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:22.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:22.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:22.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_31 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_31 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:22.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_31.status 2024-09-11T09:04:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_31.status 2024-09-11T09:04:22.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.status 2024-09-11T09:04:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_31.status 2024-09-11T09:04:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_31.status 2024-09-11T09:04:22.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.status 2024-09-11T09:04:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_31 2024-09-11T09:04:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:22.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_31 2024-09-11T09:04:22.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 146 2024-09-11T09:04:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_31 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.status 2024-09-11T09:04:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:22.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.status 2024-09-11T09:04:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_31 --format xml 2024-09-11T09:04:22.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:22.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=146 2024-09-11T09:04:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:22.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.status 2024-09-11T09:04:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:23.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.status 2024-09-11T09:04:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-11T09:04:23.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:23.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:23.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 146 = '' 2024-09-11T09:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:23.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.status 2024-09-11T09:04:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:23.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.status 2024-09-11T09:04:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-11T09:04:23.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:23.605 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:23.606 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:23.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 146 = '' 2024-09-11T09:04:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:23.609 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:24.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_31 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.status 2024-09-11T09:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.status 2024-09-11T09:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_31 --format xml 2024-09-11T09:04:24.411 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:24.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=146 2024-09-11T09:04:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 146 = 146 2024-09-11T09:04:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_31 snap 2024-09-11T09:04:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:24.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:24.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:25.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_31@snap 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_31': 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12a1ec447598 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12a1ec447598 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:25.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:08 2024 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:08 2024 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:08 2024 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a8e3428e-5e9d-4305-a12d-e6118689e825 2024-09-11T09:04:25.548 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_31 snap 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.export 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.export 2024-09-11T09:04:25.552 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.export 2024-09-11T09:04:25.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_31@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.export 2024-09-11T09:04:25.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:25.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_31@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.export 2024-09-11T09:04:26.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:26.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.export 2024-09-11T09:04:26.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_31.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_31.export 2024-09-11T09:04:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_32 2024-09-11T09:04:26.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_32 snap 2024-09-11T09:04:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:26.172 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_32@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_32 2024-09-11T09:04:27.769 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_32 started 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:27.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_32 started 2024-09-11T09:04:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:28.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_32 2024-09-11T09:04:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:28.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:28.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_32 2024-09-11T09:04:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:28.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:28.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:28.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:28.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:28.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_32 2024-09-11T09:04:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-11T09:04:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:28.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_32 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:28.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:28.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:28.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:28.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:28.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:28.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:28.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_32 2024-09-11T09:04:29.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-11T09:04:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:29.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:29.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_32 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:04:29.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:29.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:04:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:04:29.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:04:29.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:29.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_32 2024-09-11T09:04:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_32'] 2024-09-11T09:04:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:29.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_32 2024-09-11T09:04:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:04:29.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:29.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:04:29.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:04:29.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:29.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_32 2024-09-11T09:04:29.340 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:29.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:29.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:29.344 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:29.347 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_32 2024-09-11T09:04:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:29.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_32 2024-09-11T09:04:29.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_32.status 2024-09-11T09:04:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_32.status 2024-09-11T09:04:29.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.status 2024-09-11T09:04:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_32.status 2024-09-11T09:04:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_32.status 2024-09-11T09:04:29.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.status 2024-09-11T09:04:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_32 2024-09-11T09:04:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:29.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:29.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_32 2024-09-11T09:04:29.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 148 2024-09-11T09:04:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_32 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.status 2024-09-11T09:04:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:29.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.status 2024-09-11T09:04:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_32 --format xml 2024-09-11T09:04:29.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:29.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=148 2024-09-11T09:04:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:29.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:30.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.status 2024-09-11T09:04:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.status 2024-09-11T09:04:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-11T09:04:30.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:30.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:30.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 148 = '' 2024-09-11T09:04:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:30.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.status 2024-09-11T09:04:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:30.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.status 2024-09-11T09:04:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-11T09:04:30.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:30.594 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:30.595 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:30.597 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 148 = '' 2024-09-11T09:04:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:30.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_32 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.status 2024-09-11T09:04:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.status 2024-09-11T09:04:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_32 --format xml 2024-09-11T09:04:31.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:31.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=148 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 148 = 148 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_32 snap 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:31.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:31.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:32.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_32@snap 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_32': 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12a1cb7e51bf 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12a1cb7e51bf 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:32.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:19 2024 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:19 2024 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:19 2024 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 62829279-46f0-42ff-8ae0-7092ec6a74d0 2024-09-11T09:04:32.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_32 snap 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.export 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.export 2024-09-11T09:04:32.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.export 2024-09-11T09:04:32.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_32@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.export 2024-09-11T09:04:32.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:32.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_32@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.export 2024-09-11T09:04:33.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:33.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.export 2024-09-11T09:04:33.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_32.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_32.export 2024-09-11T09:04:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_33 2024-09-11T09:04:33.174 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_33 snap 2024-09-11T09:04:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:33.175 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_33@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_33 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_33 started 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:34.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:35.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_33 started 2024-09-11T09:04:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:35.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_33 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:35.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:35.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_33 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:35.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:35.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:35.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:35.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:35.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:35.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_33 2024-09-11T09:04:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_33'] 2024-09-11T09:04:35.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:35.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:35.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_33 2024-09-11T09:04:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:35.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:35.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:35.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:35.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:35.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:35.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_33 2024-09-11T09:04:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:36.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:36.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_33 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_33 2024-09-11T09:04:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_33.status 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_33.status 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.status 2024-09-11T09:04:36.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_33.status 2024-09-11T09:04:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_33.status 2024-09-11T09:04:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.status 2024-09-11T09:04:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_33 2024-09-11T09:04:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:36.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_33 2024-09-11T09:04:36.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 150 2024-09-11T09:04:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_33 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.status 2024-09-11T09:04:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:36.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.status 2024-09-11T09:04:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_33 --format xml 2024-09-11T09:04:36.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:36.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=150 2024-09-11T09:04:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:36.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:37.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.status 2024-09-11T09:04:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.status 2024-09-11T09:04:37.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-11T09:04:37.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:37.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:37.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:37.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 150 = '' 2024-09-11T09:04:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:37.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.status 2024-09-11T09:04:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:37.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.status 2024-09-11T09:04:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-11T09:04:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:37.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:37.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 150 = '' 2024-09-11T09:04:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:37.642 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_33 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.status 2024-09-11T09:04:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:38.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.status 2024-09-11T09:04:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_33 --format xml 2024-09-11T09:04:38.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:38.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=150 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 150 = 150 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_33 snap 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:38.503 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:38.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:39.504 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_33@snap 2024-09-11T09:04:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_33': 2024-09-11T09:04:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:39.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10adcedb28dc 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10adcedb28dc 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:28 2024 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:28 2024 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:28 2024 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:39.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 3b06c806-6a3f-42c2-bb4e-e1f16e05e5d7 2024-09-11T09:04:39.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_33 snap 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.export 2024-09-11T09:04:39.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.export 2024-09-11T09:04:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.export 2024-09-11T09:04:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_33@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.export 2024-09-11T09:04:39.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_33@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.export 2024-09-11T09:04:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:40.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.export 2024-09-11T09:04:40.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_33.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_33.export 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_34 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_34 snap 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:40.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_34@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_34 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_34 started 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:41.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:41.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_34 started 2024-09-11T09:04:42.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:42.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_34 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:42.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:42.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:42.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_34 2024-09-11T09:04:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:42.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:42.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:42.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:42.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:42.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:42.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_34 2024-09-11T09:04:42.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_34'] 2024-09-11T09:04:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:42.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:42.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_34 2024-09-11T09:04:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:42.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:42.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:42.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:42.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:42.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:42.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_34 2024-09-11T09:04:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_34 2024-09-11T09:04:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_34 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_34.status 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_34.status 2024-09-11T09:04:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.status 2024-09-11T09:04:43.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_34.status 2024-09-11T09:04:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_34.status 2024-09-11T09:04:43.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.status 2024-09-11T09:04:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_34 2024-09-11T09:04:43.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:43.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_34 2024-09-11T09:04:43.864 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T09:04:43.867+0000 7f00d11de640 -1 reset not still connected to 0x55b5dc3c7e10 2024-09-11T09:04:43.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 152 2024-09-11T09:04:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_34 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.status 2024-09-11T09:04:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.status 2024-09-11T09:04:43.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_34 --format xml 2024-09-11T09:04:43.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:43.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=152 2024-09-11T09:04:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:43.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.status 2024-09-11T09:04:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:44.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.status 2024-09-11T09:04:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-11T09:04:44.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:44.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:44.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:44.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 152 = '' 2024-09-11T09:04:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:44.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:44.405 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T09:04:44.406 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T09:04:44.406 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T09:04:44.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:04:44.407+0000 7f88d1b53640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 45250) UID: 1000 2024-09-11T09:04:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.status 2024-09-11T09:04:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:44.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.status 2024-09-11T09:04:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-11T09:04:44.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:44.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:44.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 152 = '' 2024-09-11T09:04:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:44.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:45.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_34 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.status 2024-09-11T09:04:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.status 2024-09-11T09:04:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_34 --format xml 2024-09-11T09:04:45.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:45.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=152 2024-09-11T09:04:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 152 = 152 2024-09-11T09:04:45.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_34 snap 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:45.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:46.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_34@snap 2024-09-11T09:04:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_34': 2024-09-11T09:04:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 128c3e0de73d 2024-09-11T09:04:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.128c3e0de73d 2024-09-11T09:04:46.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:43 2024 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:43 2024 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:43 2024 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 75f0a352-e3d0-46c7-8e9c-cd21648d227e 2024-09-11T09:04:46.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_34 snap 2024-09-11T09:04:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:04:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:46.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.export 2024-09-11T09:04:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.export 2024-09-11T09:04:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.export 2024-09-11T09:04:46.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_34@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.export 2024-09-11T09:04:46.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:46.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_34@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.export 2024-09-11T09:04:47.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:47.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.export 2024-09-11T09:04:47.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_34.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_34.export 2024-09-11T09:04:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:47.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_35 2024-09-11T09:04:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_35 snap 2024-09-11T09:04:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:47.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_35@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_35 2024-09-11T09:04:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_35 started 2024-09-11T09:04:48.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:48.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_35 started 2024-09-11T09:04:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:49.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:49.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_35 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:49.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:49.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_35 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:49.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:49.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:49.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:49.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:49.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:49.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_35 2024-09-11T09:04:50.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: exception getting command descriptions: [Errno 2] No such file or directory 2024-09-11T09:04:50.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_35 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:50.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:50.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:50.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:50.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:50.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_35 2024-09-11T09:04:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_35'] 2024-09-11T09:04:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:50.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_35 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:50.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:04:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:04:50.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:04:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:04:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:50.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:04:50.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:04:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:50.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:50.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_35 2024-09-11T09:04:50.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:50.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:50.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:50.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:50.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_35 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_35 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:50.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_35.status 2024-09-11T09:04:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_35.status 2024-09-11T09:04:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:50.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.status 2024-09-11T09:04:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_35.status 2024-09-11T09:04:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_35.status 2024-09-11T09:04:50.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.status 2024-09-11T09:04:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_35 2024-09-11T09:04:50.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:50.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_35 2024-09-11T09:04:50.406 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T09:04:50.407 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:04:50.407 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T09:04:50.407 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T09:04:50.407 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T09:04:50.410 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T09:04:50.410 INFO:tasks.rbd_mirror_thrash:waiting for 103 secs before thrashing 2024-09-11T09:04:50.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 154 2024-09-11T09:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_35 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.status 2024-09-11T09:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.status 2024-09-11T09:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_35 --format xml 2024-09-11T09:04:50.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:50.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:50.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=154 2024-09-11T09:04:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:50.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.status 2024-09-11T09:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:51.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.status 2024-09-11T09:04:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-11T09:04:51.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:51.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:51.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:51.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 154 = '' 2024-09-11T09:04:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:51.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.status 2024-09-11T09:04:51.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.status 2024-09-11T09:04:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-11T09:04:51.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:51.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:51.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:51.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 154 = '' 2024-09-11T09:04:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:51.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_35 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.status 2024-09-11T09:04:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:52.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.status 2024-09-11T09:04:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_35 --format xml 2024-09-11T09:04:52.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:52.570 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=154 2024-09-11T09:04:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 154 = 154 2024-09-11T09:04:52.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_35 snap 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:52.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:53.574 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_35@snap 2024-09-11T09:04:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_35': 2024-09-11T09:04:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:04:53.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1294dd7495c6 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1294dd7495c6 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:56:54 2024 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:56:54 2024 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:56:54 2024 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:04:53.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:04:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: d2627e7d-0b6b-467d-bdbf-be0af065f5f6 2024-09-11T09:04:53.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_35 snap 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.export 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.export 2024-09-11T09:04:53.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.export 2024-09-11T09:04:53.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_35@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.export 2024-09-11T09:04:53.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:53.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_35@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.export 2024-09-11T09:04:54.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:04:54.212 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.export 2024-09-11T09:04:54.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_35.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_35.export 2024-09-11T09:04:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:04:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_36 2024-09-11T09:04:54.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_36 snap 2024-09-11T09:04:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:54.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_36@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_36 2024-09-11T09:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_36 started 2024-09-11T09:04:55.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:55.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:04:56.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_36 started 2024-09-11T09:04:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:04:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:04:56.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_36 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:04:56.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:04:56.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:04:56.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:04:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_36 2024-09-11T09:04:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:04:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:04:56.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:56.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:04:56.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:04:56.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_36 2024-09-11T09:04:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-11T09:04:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:57.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_36 2024-09-11T09:04:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:04:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:04:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:04:57.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:57.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:57.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:04:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:04:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:04:57.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:04:57.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_36 2024-09-11T09:04:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-11T09:04:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:57.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:57.182 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_36 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:57.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:04:57.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:04:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:04:57.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:04:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:04:57.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_36 2024-09-11T09:04:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_36'] 2024-09-11T09:04:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:04:57.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:04:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:04:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_36 2024-09-11T09:04:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:04:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:04:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:04:57.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:04:57.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:57.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:04:57.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:04:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:04:57.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:04:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:04:57.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_36 2024-09-11T09:04:57.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:04:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:04:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:04:57.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:04:57.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:04:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:04:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:04:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_36 2024-09-11T09:04:57.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_36 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_36.status 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_36.status 2024-09-11T09:04:57.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.status 2024-09-11T09:04:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_36.status 2024-09-11T09:04:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_36.status 2024-09-11T09:04:57.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:04:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.status 2024-09-11T09:04:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_36 2024-09-11T09:04:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:57.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_36 2024-09-11T09:04:57.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 156 2024-09-11T09:04:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_36 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.status 2024-09-11T09:04:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:04:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:57.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.status 2024-09-11T09:04:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_36 --format xml 2024-09-11T09:04:57.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:57.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:04:57.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=156 2024-09-11T09:04:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:57.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:04:58.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.status 2024-09-11T09:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.status 2024-09-11T09:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-11T09:04:58.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:58.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 156 = '' 2024-09-11T09:04:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:58.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:04:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.status 2024-09-11T09:04:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.status 2024-09-11T09:04:58.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-11T09:04:58.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:04:58.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:58.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:04:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:04:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:04:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:04:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 156 = '' 2024-09-11T09:04:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:04:58.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:04:59.540 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_36 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.status 2024-09-11T09:04:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:59.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.status 2024-09-11T09:04:59.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_36 --format xml 2024-09-11T09:04:59.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:04:59.604 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:04:59.607 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=156 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 156 = 156 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_36 snap 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:04:59.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:00.608 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_36@snap 2024-09-11T09:05:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_36': 2024-09-11T09:05:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:00.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12a1398b6f5a 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12a1398b6f5a 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:04 2024 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:04 2024 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:04 2024 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: f6933439-7dab-4216-b4b5-b329ed22c0f6 2024-09-11T09:05:00.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_36 snap 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.export 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.export 2024-09-11T09:05:00.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.export 2024-09-11T09:05:00.675 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_36@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.export 2024-09-11T09:05:00.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:00.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_36@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.export 2024-09-11T09:05:01.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:01.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.export 2024-09-11T09:05:01.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_36.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_36.export 2024-09-11T09:05:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:01.270 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_37 2024-09-11T09:05:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_37 snap 2024-09-11T09:05:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:01.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_37@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_37 2024-09-11T09:05:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:02.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_37 started 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:02.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_37 started 2024-09-11T09:05:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:03.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_37 2024-09-11T09:05:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:03.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:03.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:03.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_37 2024-09-11T09:05:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:03.950 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:03.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:03.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:03.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:03.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_37 2024-09-11T09:05:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-11T09:05:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:04.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:04.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_37 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:04.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:04.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:05:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:05:04.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:05:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:04.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:04.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_37 2024-09-11T09:05:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_37'] 2024-09-11T09:05:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:04.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_37 2024-09-11T09:05:04.220 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:05:04.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:04.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:05:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:05:04.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:05:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:04.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_37 2024-09-11T09:05:04.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:04.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:04.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:04.376 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:04.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_37 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_37 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:04.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_37.status 2024-09-11T09:05:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_37.status 2024-09-11T09:05:04.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:04.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.status 2024-09-11T09:05:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_37.status 2024-09-11T09:05:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_37.status 2024-09-11T09:05:04.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_37 2024-09-11T09:05:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:04.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_37 2024-09-11T09:05:04.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 158 2024-09-11T09:05:04.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_37 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.status 2024-09-11T09:05:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.status 2024-09-11T09:05:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_37 --format xml 2024-09-11T09:05:04.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:05.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:05:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=158 2024-09-11T09:05:05.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:05.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:05.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T09:05:05.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:05.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=71 2024-09-11T09:05:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 158 = 71 2024-09-11T09:05:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:05.305 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:05:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:05.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T09:05:05.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:05.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=71 2024-09-11T09:05:05.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 158 = 71 2024-09-11T09:05:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:05.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:05:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:06.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T09:05:06.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:06.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:06.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:06.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 158 = '' 2024-09-11T09:05:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:06.647 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:05:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_37 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:08.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.status 2024-09-11T09:05:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:08.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_37 --format xml 2024-09-11T09:05:08.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:08.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=158 2024-09-11T09:05:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 158 = 158 2024-09-11T09:05:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_37 snap 2024-09-11T09:05:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:08.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:05:08.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:09.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_37@snap 2024-09-11T09:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_37': 2024-09-11T09:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ad9c894eb1 2024-09-11T09:05:09.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ad9c894eb1 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:18 2024 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:18 2024 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:18 2024 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: a953381e-b730-4277-a246-b871eee2718e 2024-09-11T09:05:09.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:09.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_37 snap 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.export 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.export 2024-09-11T09:05:09.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.export 2024-09-11T09:05:09.384 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_37@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.export 2024-09-11T09:05:09.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:09.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_37@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.export 2024-09-11T09:05:10.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:10.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.export 2024-09-11T09:05:10.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_37.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_37.export 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_38 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_38 snap 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:10.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_38@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:11.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_38 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_38 started 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:11.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:11.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_38 started 2024-09-11T09:05:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:12.958 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:12.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_38 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:12.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:12.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:12.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_38 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:12.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:12.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:12.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:12.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_38 2024-09-11T09:05:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_38'] 2024-09-11T09:05:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:13.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:13.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_38 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:13.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:05:13.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:05:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:05:13.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:13.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:05:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:05:13.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:05:13.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:13.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_38 2024-09-11T09:05:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:13.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:13.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:13.249 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_38 2024-09-11T09:05:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:13.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_38 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:13.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_38.status 2024-09-11T09:05:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_38.status 2024-09-11T09:05:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.status 2024-09-11T09:05:13.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_38.status 2024-09-11T09:05:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_38.status 2024-09-11T09:05:13.253 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:13.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_38 2024-09-11T09:05:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:13.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_38 2024-09-11T09:05:13.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 160 2024-09-11T09:05:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_38 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.status 2024-09-11T09:05:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:13.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.status 2024-09-11T09:05:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:13.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_38 --format xml 2024-09-11T09:05:14.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:05:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=160 2024-09-11T09:05:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:14.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:05:14.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:14.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T09:05:14.252 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:14.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=72 2024-09-11T09:05:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 160 = 72 2024-09-11T09:05:14.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:14.532 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:05:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:14.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T09:05:14.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:15.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:15.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:15.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 160 = '' 2024-09-11T09:05:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:15.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:05:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:15.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T09:05:15.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:15.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:15.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 160 = '' 2024-09-11T09:05:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:15.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:05:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_38 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.status 2024-09-11T09:05:17.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_38 --format xml 2024-09-11T09:05:17.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:17.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=160 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 160 = 160 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_38 snap 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:17.555 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:05:17.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:18.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_38@snap 2024-09-11T09:05:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_38': 2024-09-11T09:05:18.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12b06b53c7b4 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12b06b53c7b4 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:29 2024 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:29 2024 2024-09-11T09:05:18.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:29 2024 2024-09-11T09:05:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: e269182c-689f-4c97-bd46-05795a725cb4 2024-09-11T09:05:18.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:18.622 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_38 snap 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.export 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.export 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.export 2024-09-11T09:05:18.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_38@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.export 2024-09-11T09:05:18.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:18.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_38@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.export 2024-09-11T09:05:19.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:19.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.export 2024-09-11T09:05:19.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_38.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_38.export 2024-09-11T09:05:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_39 2024-09-11T09:05:19.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_39 snap 2024-09-11T09:05:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:19.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_39@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:20.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_39 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_39 started 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:20.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_39 started 2024-09-11T09:05:21.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:21.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_39 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:21.661 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:21.662 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_39 2024-09-11T09:05:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:21.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:21.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:21.666 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:21.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:21.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:21.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_39 2024-09-11T09:05:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-11T09:05:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:21.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_39 2024-09-11T09:05:21.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:05:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:05:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:05:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:21.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:05:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:05:21.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:05:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:05:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:05:21.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:05:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:21.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:05:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:05:21.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:05:21.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:21.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_39 2024-09-11T09:05:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_39'] 2024-09-11T09:05:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:21.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_39 2024-09-11T09:05:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:05:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:21.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:05:21.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:21.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:05:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:05:21.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:05:21.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:21.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_39 2024-09-11T09:05:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:22.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:22.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:22.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_39 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_39 2024-09-11T09:05:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_39.status 2024-09-11T09:05:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_39.status 2024-09-11T09:05:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.status 2024-09-11T09:05:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_39.status 2024-09-11T09:05:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_39.status 2024-09-11T09:05:22.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.status 2024-09-11T09:05:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_39 2024-09-11T09:05:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:22.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_39 2024-09-11T09:05:22.664 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 162 2024-09-11T09:05:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_39 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.status 2024-09-11T09:05:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:22.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.status 2024-09-11T09:05:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_39 --format xml 2024-09-11T09:05:22.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:22.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:05:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=162 2024-09-11T09:05:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:22.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:05:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.status 2024-09-11T09:05:22.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.status 2024-09-11T09:05:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-11T09:05:22.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:23.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=73 2024-09-11T09:05:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 162 = 73 2024-09-11T09:05:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:23.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:05:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.status 2024-09-11T09:05:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.status 2024-09-11T09:05:23.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-11T09:05:23.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:23.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:23.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:23.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 162 = '' 2024-09-11T09:05:23.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:23.478 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:05:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_39 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.status 2024-09-11T09:05:24.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.status 2024-09-11T09:05:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_39 --format xml 2024-09-11T09:05:24.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:24.356 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=162 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 162 = 162 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_39 snap 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:24.360 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:05:24.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:25.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_39@snap 2024-09-11T09:05:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_39': 2024-09-11T09:05:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:25.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12b041322e92 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12b041322e92 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:38 2024 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:38 2024 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:38 2024 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: c046011c-40fa-4138-bba6-2fc652d532ec 2024-09-11T09:05:25.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:25.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_39 snap 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.export 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.export 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.export 2024-09-11T09:05:25.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_39@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.export 2024-09-11T09:05:25.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:25.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_39@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.export 2024-09-11T09:05:26.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:26.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.export 2024-09-11T09:05:26.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_39.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_39.export 2024-09-11T09:05:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_40 2024-09-11T09:05:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_40 snap 2024-09-11T09:05:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:26.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:26.339 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_40@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_40 2024-09-11T09:05:27.684 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_40 started 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:27.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_40 started 2024-09-11T09:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:28.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:28.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_40 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:28.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:28.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_40 2024-09-11T09:05:28.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:28.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:28.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:28.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:28.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:28.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_40 2024-09-11T09:05:28.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_40 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:28.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:05:28.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:05:28.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:05:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:28.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_40 2024-09-11T09:05:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_40'] 2024-09-11T09:05:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:28.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_40 2024-09-11T09:05:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:05:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:05:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:05:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:05:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:05:28.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:05:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:29.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_40 2024-09-11T09:05:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:29.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:29.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:29.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_40 2024-09-11T09:05:29.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_40 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_40.status 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:29.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_40.status 2024-09-11T09:05:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.status 2024-09-11T09:05:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_40.status 2024-09-11T09:05:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_40.status 2024-09-11T09:05:29.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_40 2024-09-11T09:05:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:29.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_40 2024-09-11T09:05:29.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 164 2024-09-11T09:05:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_40 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.status 2024-09-11T09:05:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.status 2024-09-11T09:05:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:29.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_40 --format xml 2024-09-11T09:05:29.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:05:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=164 2024-09-11T09:05:29.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:29.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:05:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:29.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T09:05:29.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=74 2024-09-11T09:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 164 = 74 2024-09-11T09:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:30.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:05:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T09:05:30.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:30.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:30.508 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:30.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 164 = '' 2024-09-11T09:05:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:30.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:05:31.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T09:05:31.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:31.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:31.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:31.412 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 164 = '' 2024-09-11T09:05:31.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:31.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:05:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_40 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:33.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.status 2024-09-11T09:05:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_40 --format xml 2024-09-11T09:05:33.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:33.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:33.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=164 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 164 = 164 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_40 snap 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:05:33.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:34.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_40@snap 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_40': 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12b094bad470 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12b094bad470 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:34.146 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:57:47 2024 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:57:47 2024 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:57:47 2024 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 24872562-d2cb-4569-88af-e8992d096531 2024-09-11T09:05:34.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:34.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_40 snap 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.export 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.export 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.export 2024-09-11T09:05:34.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_40@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.export 2024-09-11T09:05:34.361 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:34.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_40@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.export 2024-09-11T09:05:34.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:34.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.export 2024-09-11T09:05:34.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_40.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_40.export 2024-09-11T09:05:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:34.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_41 2024-09-11T09:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_41 snap 2024-09-11T09:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:34.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_41@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_41 2024-09-11T09:05:36.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_41 started 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:36.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_41 started 2024-09-11T09:05:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:37.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_41 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:37.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:37.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_41 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:37.737 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:37.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:37.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:37.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_41 2024-09-11T09:05:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-11T09:05:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:37.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_41 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:37.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:37.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:05:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:05:37.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:37.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_41 2024-09-11T09:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-11T09:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:38.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_41 2024-09-11T09:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:05:38.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:05:38.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:38.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:38.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:05:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:05:38.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:05:38.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:05:38.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_41 2024-09-11T09:05:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_41'] 2024-09-11T09:05:38.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:38.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_41 2024-09-11T09:05:38.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:05:38.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:38.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:05:38.169 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:05:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:05:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:05:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:05:38.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_41 2024-09-11T09:05:38.319 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:38.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:38.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:38.323 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_41 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:38.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_41 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:38.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_41.status 2024-09-11T09:05:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_41.status 2024-09-11T09:05:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.status 2024-09-11T09:05:38.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_41.status 2024-09-11T09:05:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_41.status 2024-09-11T09:05:38.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.status 2024-09-11T09:05:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_41 2024-09-11T09:05:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:38.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:38.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_41 2024-09-11T09:05:38.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 166 2024-09-11T09:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_41 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.status 2024-09-11T09:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:38.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.status 2024-09-11T09:05:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_41 --format xml 2024-09-11T09:05:38.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:38.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:05:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=166 2024-09-11T09:05:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:38.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:05:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.status 2024-09-11T09:05:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.status 2024-09-11T09:05:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-11T09:05:39.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:39.101 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:39.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 166 = '' 2024-09-11T09:05:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:39.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:05:39.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.status 2024-09-11T09:05:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.status 2024-09-11T09:05:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-11T09:05:39.507 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:39.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:39.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 166 = '' 2024-09-11T09:05:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:39.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:05:40.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_41 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.status 2024-09-11T09:05:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.status 2024-09-11T09:05:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_41 --format xml 2024-09-11T09:05:40.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:40.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=166 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 166 = 166 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_41 snap 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:40.450 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:05:40.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:41.451 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_41@snap 2024-09-11T09:05:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_41': 2024-09-11T09:05:41.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ad197c8da1 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ad197c8da1 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:00 2024 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:00 2024 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:00 2024 2024-09-11T09:05:41.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: e18f305c-3600-4a05-ac01-cbd0ec5179dd 2024-09-11T09:05:41.512 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_41 snap 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.export 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.export 2024-09-11T09:05:41.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.export 2024-09-11T09:05:41.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_41@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.export 2024-09-11T09:05:41.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:41.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_41@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.export 2024-09-11T09:05:41.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:41.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.export 2024-09-11T09:05:42.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_41.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_41.export 2024-09-11T09:05:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:42.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_42 2024-09-11T09:05:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_42 snap 2024-09-11T09:05:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:42.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_42@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_42 2024-09-11T09:05:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:43.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_42 started 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:43.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_42 started 2024-09-11T09:05:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:44.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_42 2024-09-11T09:05:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:44.752 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:44.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_42 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:44.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:44.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:44.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:44.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:44.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:44.783 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_42 2024-09-11T09:05:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:44.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:44.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:44.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_42 2024-09-11T09:05:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:44.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_42 2024-09-11T09:05:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:44.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_42.status 2024-09-11T09:05:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_42.status 2024-09-11T09:05:44.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.status 2024-09-11T09:05:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_42.status 2024-09-11T09:05:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_42.status 2024-09-11T09:05:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.status 2024-09-11T09:05:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_42 2024-09-11T09:05:44.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:44.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_42 2024-09-11T09:05:45.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 168 2024-09-11T09:05:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_42 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.status 2024-09-11T09:05:45.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.status 2024-09-11T09:05:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:45.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_42 --format xml 2024-09-11T09:05:45.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:05:45.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=168 2024-09-11T09:05:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:45.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.status 2024-09-11T09:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.status 2024-09-11T09:05:46.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-11T09:05:46.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:46.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:46.118 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 168 = '' 2024-09-11T09:05:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:46.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:05:46.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.status 2024-09-11T09:05:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.status 2024-09-11T09:05:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-11T09:05:46.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:46.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:46.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 168 = '' 2024-09-11T09:05:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:46.592 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:05:47.311 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T09:05:47.311 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T09:05:47.311 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T09:05:47.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:05:47.314+0000 7fe586617640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 48317) UID: 1000 2024-09-11T09:05:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_42 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.status 2024-09-11T09:05:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.status 2024-09-11T09:05:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_42 --format xml 2024-09-11T09:05:47.394 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:47.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=168 2024-09-11T09:05:47.461 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 168 = 168 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_42 snap 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:05:47.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:48.462 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_42@snap 2024-09-11T09:05:48.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_42': 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1294303afc13 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1294303afc13 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:12 2024 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:12 2024 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:12 2024 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:48.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 52286b49-eaf5-465a-988f-e2ac13d2e71c 2024-09-11T09:05:48.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_42 snap 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.export 2024-09-11T09:05:48.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.export 2024-09-11T09:05:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.export 2024-09-11T09:05:48.584 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_42@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.export 2024-09-11T09:05:48.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:48.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_42@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.export 2024-09-11T09:05:49.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:49.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.export 2024-09-11T09:05:49.272 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_42.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_42.export 2024-09-11T09:05:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_43 2024-09-11T09:05:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_43 snap 2024-09-11T09:05:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_43@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_43 2024-09-11T09:05:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:50.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_43 started 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:50.807 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_43 started 2024-09-11T09:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:51.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_43 2024-09-11T09:05:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:51.809 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:51.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_43 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:51.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:51.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:51.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:51.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_43 2024-09-11T09:05:51.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:51.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:51.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:51.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_43 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_43 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:51.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_43.status 2024-09-11T09:05:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:51.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_43.status 2024-09-11T09:05:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.status 2024-09-11T09:05:51.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_43.status 2024-09-11T09:05:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_43.status 2024-09-11T09:05:51.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.status 2024-09-11T09:05:51.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_43 2024-09-11T09:05:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:51.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_43 2024-09-11T09:05:52.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 170 2024-09-11T09:05:52.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_43 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.status 2024-09-11T09:05:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.status 2024-09-11T09:05:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_43 --format xml 2024-09-11T09:05:52.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:52.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:05:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=170 2024-09-11T09:05:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:52.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:05:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.status 2024-09-11T09:05:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.status 2024-09-11T09:05:53.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-11T09:05:53.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:53.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:53.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 170 = '' 2024-09-11T09:05:53.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:53.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:05:53.313 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T09:05:53.313 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:05:53.313 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T09:05:53.313 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T09:05:53.314 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T09:05:53.317 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T09:05:53.317 INFO:tasks.rbd_mirror_thrash:waiting for 97 secs before thrashing 2024-09-11T09:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.status 2024-09-11T09:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.status 2024-09-11T09:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-11T09:05:53.559 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:53.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:05:53.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:05:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:05:53.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:05:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:05:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 170 = '' 2024-09-11T09:05:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:05:53.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:05:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_43 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.status 2024-09-11T09:05:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:54.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.status 2024-09-11T09:05:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_43 --format xml 2024-09-11T09:05:54.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:05:54.493 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:05:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=170 2024-09-11T09:05:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 170 = 170 2024-09-11T09:05:54.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_43 snap 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:05:54.497 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:55.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_43@snap 2024-09-11T09:05:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_43': 2024-09-11T09:05:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:05:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:05:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:05:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12a180808ca6 2024-09-11T09:05:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12a180808ca6 2024-09-11T09:05:55.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:18 2024 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:18 2024 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:18 2024 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 540fcbee-d8ed-438b-8b66-b3b65770db9f 2024-09-11T09:05:55.562 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:05:55.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_43 snap 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.export 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.export 2024-09-11T09:05:55.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.export 2024-09-11T09:05:55.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_43@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.export 2024-09-11T09:05:55.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:55.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_43@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.export 2024-09-11T09:05:56.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:05:56.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.export 2024-09-11T09:05:56.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_43.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_43.export 2024-09-11T09:05:56.179 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_44 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_44 snap 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:05:56.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_44@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_44 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_44 started 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:57.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:05:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:05:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:57.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:05:58.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_44 started 2024-09-11T09:05:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:05:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:05:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:05:58.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_44 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:05:58.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:05:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:05:58.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:05:58.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:05:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_44 2024-09-11T09:05:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:05:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:05:58.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:58.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:58.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:05:58.828 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:05:58.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:05:58.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_44 2024-09-11T09:05:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_44'] 2024-09-11T09:05:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:05:58.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_44 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:05:58.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:05:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:05:58.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:58.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:05:58.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:05:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:05:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:05:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:05:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:05:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:05:58.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:58.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:05:58.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_44 2024-09-11T09:05:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:05:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:05:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:05:59.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:05:59.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:05:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:05:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:05:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:05:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_44 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_44 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_44.status 2024-09-11T09:05:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_44.status 2024-09-11T09:05:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.status 2024-09-11T09:05:59.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_44.status 2024-09-11T09:05:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_44.status 2024-09-11T09:05:59.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:05:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.status 2024-09-11T09:05:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_44 2024-09-11T09:05:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:59.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:59.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_44 2024-09-11T09:05:59.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 172 2024-09-11T09:05:59.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_44 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.status 2024-09-11T09:05:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:05:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:05:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:05:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.status 2024-09-11T09:05:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_44 --format xml 2024-09-11T09:05:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:00.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:06:00.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=172 2024-09-11T09:06:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:00.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.status 2024-09-11T09:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:00.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.status 2024-09-11T09:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-11T09:06:00.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:00.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:00.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 172 = '' 2024-09-11T09:06:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:00.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:06:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.status 2024-09-11T09:06:00.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:06:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.status 2024-09-11T09:06:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-11T09:06:00.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:00.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:00.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:00.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 172 = '' 2024-09-11T09:06:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:00.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:06:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_44 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.status 2024-09-11T09:06:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:01.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:06:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.status 2024-09-11T09:06:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_44 --format xml 2024-09-11T09:06:01.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:01.741 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=172 2024-09-11T09:06:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 172 = 172 2024-09-11T09:06:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:01.746 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_44 snap 2024-09-11T09:06:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:06:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:06:01.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:02.747 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_44@snap 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_44': 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12b0f292d860 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12b0f292d860 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:06:02.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:24 2024 2024-09-11T09:06:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:24 2024 2024-09-11T09:06:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:24 2024 2024-09-11T09:06:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:06:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:06:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:06:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 9eba4b2e-6f4b-4273-9365-f3717576509c 2024-09-11T09:06:02.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:06:02.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_44 snap 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.export 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.export 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.export 2024-09-11T09:06:02.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_44@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.export 2024-09-11T09:06:03.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:03.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_44@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.export 2024-09-11T09:06:03.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:03.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.export 2024-09-11T09:06:03.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_44.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_44.export 2024-09-11T09:06:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_45 2024-09-11T09:06:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_45 snap 2024-09-11T09:06:03.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:03.410 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_45@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:06:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_45 2024-09-11T09:06:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:04.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_45 started 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:04.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_45 started 2024-09-11T09:06:05.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:06:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:06:05.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_45 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:06:05.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:06:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:06:05.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:06:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_45 2024-09-11T09:06:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:06:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:06:05.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:05.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:06:05.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:06:05.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:06:05.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_45 2024-09-11T09:06:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-11T09:06:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_45 2024-09-11T09:06:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:06:06.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:06:06.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:06.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:06:06.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:06:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:06.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:06.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_45 2024-09-11T09:06:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-11T09:06:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:06.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_45 2024-09-11T09:06:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:06:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:06.147 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:06:06.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:06:06.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:06:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:06:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:06.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_45 2024-09-11T09:06:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_45'] 2024-09-11T09:06:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:06.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_45 2024-09-11T09:06:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:06:06.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:06:06.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:06:06.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:06:06.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:06:06.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:06:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:06:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:06:06.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_45 2024-09-11T09:06:06.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:06:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:06:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:06:06.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:06:06.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_45 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:06.442 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_45 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_45.status 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_45.status 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:06.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.status 2024-09-11T09:06:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_45.status 2024-09-11T09:06:06.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_45.status 2024-09-11T09:06:06.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:06.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_45 2024-09-11T09:06:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:06.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_45 2024-09-11T09:06:06.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 174 2024-09-11T09:06:06.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_45 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.status 2024-09-11T09:06:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.status 2024-09-11T09:06:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_45 --format xml 2024-09-11T09:06:06.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:06.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:06:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=174 2024-09-11T09:06:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:06.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:06:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:07.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T09:06:07.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:07.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=79 2024-09-11T09:06:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 174 = 79 2024-09-11T09:06:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:07.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:06:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:07.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T09:06:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:07.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:07.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 174 = '' 2024-09-11T09:06:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:07.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:06:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:08.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:08.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:08.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T09:06:08.667 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:08.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:08.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 174 = '' 2024-09-11T09:06:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:08.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:06:10.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_45 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.status 2024-09-11T09:06:10.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_45 --format xml 2024-09-11T09:06:10.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:10.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=174 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 174 = 174 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_45 snap 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:06:10.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:11.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_45@snap 2024-09-11T09:06:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_45': 2024-09-11T09:06:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:06:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:06:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:06:11.408 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10ad84945b3a 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10ad84945b3a 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:32 2024 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:32 2024 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:32 2024 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: f5150c59-0ae1-4502-a239-7a0369bc62f5 2024-09-11T09:06:11.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_45 snap 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.export 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.export 2024-09-11T09:06:11.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.export 2024-09-11T09:06:11.414 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_45@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.export 2024-09-11T09:06:11.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:11.631 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_45@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.export 2024-09-11T09:06:11.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:11.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.export 2024-09-11T09:06:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_45.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_45.export 2024-09-11T09:06:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_46 2024-09-11T09:06:12.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_46 snap 2024-09-11T09:06:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:12.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_46@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:06:13.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_46 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_46 started 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:13.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:14.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_46 started 2024-09-11T09:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:06:14.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_46 2024-09-11T09:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:06:14.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:06:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:14.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:06:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:06:14.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:06:14.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_46 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:14.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:14.895 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:06:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:06:14.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:06:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:14.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_46 2024-09-11T09:06:15.044 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:06:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:06:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:06:15.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:06:15.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_46 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:15.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_46 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_46.status 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_46.status 2024-09-11T09:06:15.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.status 2024-09-11T09:06:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_46.status 2024-09-11T09:06:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_46.status 2024-09-11T09:06:15.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:15.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.status 2024-09-11T09:06:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_46 2024-09-11T09:06:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:15.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_46 2024-09-11T09:06:15.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 176 2024-09-11T09:06:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_46 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.status 2024-09-11T09:06:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:15.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.status 2024-09-11T09:06:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_46 --format xml 2024-09-11T09:06:15.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:15.963 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:06:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=176 2024-09-11T09:06:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:15.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:06:16.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.status 2024-09-11T09:06:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.status 2024-09-11T09:06:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-11T09:06:16.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:16.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:16.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 176 = '' 2024-09-11T09:06:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:16.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:06:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.status 2024-09-11T09:06:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:16.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.status 2024-09-11T09:06:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-11T09:06:16.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:16.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:16.708 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 176 = '' 2024-09-11T09:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:16.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:06:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_46 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.status 2024-09-11T09:06:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:17.510 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.status 2024-09-11T09:06:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_46 --format xml 2024-09-11T09:06:17.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:17.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=176 2024-09-11T09:06:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 176 = 176 2024-09-11T09:06:17.581 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_46 snap 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:06:17.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:18.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_46@snap 2024-09-11T09:06:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_46': 2024-09-11T09:06:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:06:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:06:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:06:18.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 129485e8091d 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.129485e8091d 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:42 2024 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:42 2024 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:42 2024 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8cd87261-11aa-45a3-8d2c-211e06dbeb62 2024-09-11T09:06:18.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:06:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_46 snap 2024-09-11T09:06:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:06:18.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:06:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.export 2024-09-11T09:06:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.export 2024-09-11T09:06:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.export 2024-09-11T09:06:18.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_46@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.export 2024-09-11T09:06:18.842 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:18.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_46@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.export 2024-09-11T09:06:19.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.export 2024-09-11T09:06:19.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_46.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_46.export 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_47 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_47 snap 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:19.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:19.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_47@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:06:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_47 2024-09-11T09:06:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:20.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_47 started 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:20.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_47 started 2024-09-11T09:06:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:06:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:06:21.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:06:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_47 2024-09-11T09:06:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:06:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:06:21.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:06:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:06:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:06:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:06:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:06:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:06:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_47 2024-09-11T09:06:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:06:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:21.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:06:21.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:06:21.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:06:21.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:06:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:21.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_47 2024-09-11T09:06:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-11T09:06:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:22.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_47 2024-09-11T09:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:06:22.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:06:22.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:06:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:06:22.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:06:22.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:06:22.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:22.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_47 2024-09-11T09:06:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-11T09:06:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:22.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_47 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:22.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:06:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:06:22.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:06:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:06:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:22.228 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:06:22.229 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:06:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:22.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_47 2024-09-11T09:06:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_47'] 2024-09-11T09:06:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:22.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:22.365 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:3 rbd mirror status mirror/image_47 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:3 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:3 cluster instance 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:3 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=3 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 3 = cluster1:3 ']' 2024-09-11T09:06:22.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=3 2024-09-11T09:06:22.367 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:06:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:3 cluster1 2024-09-11T09:06:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:3 2024-09-11T09:06:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:3 local_cluster instance 2024-09-11T09:06:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:3 2024-09-11T09:06:22.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=3 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 3 = cluster1:3 ']' 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=3 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=3 2024-09-11T09:06:22.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.3 'admin socket' 2024-09-11T09:06:22.390 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:06:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:06:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok 2024-09-11T09:06:22.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.3.asok rbd mirror status mirror/image_47 2024-09-11T09:06:22.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:06:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:06:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:06:22.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:06:22.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:06:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:06:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:06:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_47 2024-09-11T09:06:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:22.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_47 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_47.status 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:22.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_47.status 2024-09-11T09:06:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.status 2024-09-11T09:06:22.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_47.status 2024-09-11T09:06:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_47.status 2024-09-11T09:06:22.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.status 2024-09-11T09:06:22.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_47 2024-09-11T09:06:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:22.539 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_47 2024-09-11T09:06:22.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 178 2024-09-11T09:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_47 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.status 2024-09-11T09:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.status 2024-09-11T09:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_47 --format xml 2024-09-11T09:06:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:23.011 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:06:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=178 2024-09-11T09:06:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:23.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:06:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.status 2024-09-11T09:06:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:23.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.status 2024-09-11T09:06:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-11T09:06:23.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:23.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:23.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:23.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 178 = '' 2024-09-11T09:06:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:23.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:06:23.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.status 2024-09-11T09:06:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:23.691 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.status 2024-09-11T09:06:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-11T09:06:23.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:23.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:23.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 178 = '' 2024-09-11T09:06:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:23.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:06:24.560 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_47 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.status 2024-09-11T09:06:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.status 2024-09-11T09:06:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_47 --format xml 2024-09-11T09:06:24.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:24.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=178 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 178 = 178 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_47 snap 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:24.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:06:24.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:25.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_47@snap 2024-09-11T09:06:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_47': 2024-09-11T09:06:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:06:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:06:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:06:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12a11fd62bda 2024-09-11T09:06:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12a11fd62bda 2024-09-11T09:06:25.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:58:52 2024 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:58:52 2024 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:58:52 2024 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 8ad010f4-22cf-4eef-8e81-0a6236a13192 2024-09-11T09:06:25.697 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_47 snap 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.export 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.export 2024-09-11T09:06:25.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.export 2024-09-11T09:06:25.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_47@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.export 2024-09-11T09:06:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:25.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_47@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.export 2024-09-11T09:06:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:26.209 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.export 2024-09-11T09:06:26.285 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_47.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_47.export 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_48 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_48 snap 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:26.326 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:26.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_48@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:06:27.944 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_48 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_48 started 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:27.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_48 started 2024-09-11T09:06:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:28.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:06:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:06:28.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_48 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:06:28.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:06:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:06:28.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:06:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_48 2024-09-11T09:06:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:06:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:28.952 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:28.953 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:06:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:06:28.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:28.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_48 2024-09-11T09:06:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-11T09:06:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_48 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:06:29.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:06:29.097 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:06:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:29.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_48 2024-09-11T09:06:29.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_48'] 2024-09-11T09:06:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:29.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:29.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_48 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:29.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:29.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:06:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:06:29.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:06:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:29.254 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:29.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_48 2024-09-11T09:06:29.376 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:06:29.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:06:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:06:29.378 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:06:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:06:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:06:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:06:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_48 2024-09-11T09:06:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:29.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_48 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_48.status 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_48.status 2024-09-11T09:06:29.382 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.status 2024-09-11T09:06:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_48.status 2024-09-11T09:06:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_48.status 2024-09-11T09:06:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.status 2024-09-11T09:06:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_48 2024-09-11T09:06:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:29.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_48 2024-09-11T09:06:29.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 180 2024-09-11T09:06:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_48 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.status 2024-09-11T09:06:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:29.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.status 2024-09-11T09:06:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:29.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_48 --format xml 2024-09-11T09:06:30.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:06:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=180 2024-09-11T09:06:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:30.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.status 2024-09-11T09:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.status 2024-09-11T09:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-11T09:06:30.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:30.303 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:30.304 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:30.306 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 180 = '' 2024-09-11T09:06:30.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:30.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:06:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.status 2024-09-11T09:06:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:30.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.status 2024-09-11T09:06:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-11T09:06:30.711 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:30.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:30.777 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 180 = '' 2024-09-11T09:06:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:30.778 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:06:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_48 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.status 2024-09-11T09:06:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:31.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.status 2024-09-11T09:06:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_48 --format xml 2024-09-11T09:06:31.580 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:31.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=180 2024-09-11T09:06:31.652 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 180 = 180 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_48 snap 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:06:31.653 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:32.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_48@snap 2024-09-11T09:06:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_48': 2024-09-11T09:06:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:06:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:06:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:06:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 129491fee0c7 2024-09-11T09:06:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.129491fee0c7 2024-09-11T09:06:32.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:59:30 2024 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:59:30 2024 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:59:30 2024 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: ef84862e-99c3-4940-9190-5144ad96e521 2024-09-11T09:06:32.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_48 snap 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.export 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.export 2024-09-11T09:06:32.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.export 2024-09-11T09:06:32.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_48@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.export 2024-09-11T09:06:32.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:32.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_48@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.export 2024-09-11T09:06:33.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:33.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.export 2024-09-11T09:06:33.315 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_48.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_48.export 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_49 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_49 snap 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:33.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_49@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:06:33.411 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.8262794265487966) > weight (0.25) 2024-09-11T09:06:33.411 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.9268125972551362) > weight (0.25) 2024-09-11T09:06:33.411 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.2 with skip (0.8421346561842031) > weight (0.25) 2024-09-11T09:06:33.411 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.3 with skip (0.9612875875823624) > weight (0.25) 2024-09-11T09:06:33.412 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_49 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_49 started 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:34.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:06:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:34.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_49 started 2024-09-11T09:06:35.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:06:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:06:35.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:06:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_49 2024-09-11T09:06:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:06:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:06:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:06:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:06:35.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:06:35.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:06:35.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_49 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:06:35.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:35.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:06:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:06:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:35.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:06:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:06:35.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:06:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:36.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:36.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_49 2024-09-11T09:06:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-11T09:06:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:36.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_49 2024-09-11T09:06:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:06:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:06:36.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:06:36.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:36.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:06:36.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:06:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:36.158 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:36.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_49 2024-09-11T09:06:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_49'] 2024-09-11T09:06:36.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:36.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:36.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_49 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:06:36.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:36.280 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:06:36.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:06:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:36.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_49 2024-09-11T09:06:36.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:06:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:06:36.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:06:36.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:06:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:06:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:06:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:06:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_49 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_49 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_49.status 2024-09-11T09:06:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_49.status 2024-09-11T09:06:36.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.status 2024-09-11T09:06:36.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_49.status 2024-09-11T09:06:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_49.status 2024-09-11T09:06:36.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.status 2024-09-11T09:06:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_49 2024-09-11T09:06:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:36.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_49 2024-09-11T09:06:36.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 182 2024-09-11T09:06:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_49 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.status 2024-09-11T09:06:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:36.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.status 2024-09-11T09:06:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_49 --format xml 2024-09-11T09:06:36.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:37.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:06:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=182 2024-09-11T09:06:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:37.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:06:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.status 2024-09-11T09:06:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.status 2024-09-11T09:06:37.255 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-11T09:06:37.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:37.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:37.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=83 2024-09-11T09:06:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 182 = 83 2024-09-11T09:06:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:37.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.status 2024-09-11T09:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:37.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.status 2024-09-11T09:06:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-11T09:06:37.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:37.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:37.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:37.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:37.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 182 = '' 2024-09-11T09:06:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:37.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:06:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_49 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.status 2024-09-11T09:06:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.status 2024-09-11T09:06:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_49 --format xml 2024-09-11T09:06:38.599 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:38.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=182 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 182 = 182 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_49 snap 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:06:38.726 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:39.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_49@snap 2024-09-11T09:06:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_49': 2024-09-11T09:06:39.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 10adfe825439 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.10adfe825439 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:59:09 2024 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:59:09 2024 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:59:09 2024 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:06:39.788 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:06:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:06:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: ef6d8f68-5e32-4505-b07d-5119f8fb90c8 2024-09-11T09:06:39.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:06:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_49 snap 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.export 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.export 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.export 2024-09-11T09:06:39.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_49@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.export 2024-09-11T09:06:40.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:40.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_49@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.export 2024-09-11T09:06:40.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:40.369 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.export 2024-09-11T09:06:40.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_49.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_49.export 2024-09-11T09:06:40.488 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_50 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_snap cluster2 mirror image_50 snap 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:40.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror snap create image_50@snap --debug-rbd=20 --debug-journaler=20 2024-09-11T09:06:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_started cluster1 mirror image_50 2024-09-11T09:06:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:41.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_replay_state cluster1 mirror image_50 started 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=started 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:41.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_replay_state cluster1 mirror image_50 started 2024-09-11T09:06:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=started 2024-09-11T09:06:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_result 2024-09-11T09:06:42.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=stopped 2024-09-11T09:06:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemons cluster1 rbd mirror status mirror/image_50 2024-09-11T09:06:42.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -i state 2024-09-11T09:06:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_instance=cluster1 2024-09-11T09:06:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:06:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance=cluster1 2024-09-11T09:06:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local loop_instance 2024-09-11T09:06:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for s in 0 1 2 4 8 8 8 8 8 8 8 8 16 16 2024-09-11T09:06:42.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sleep 0 2024-09-11T09:06:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' cluster1 '!=' cluster1 ']' 2024-09-11T09:06:42.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ seq 0 3 2024-09-11T09:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:0 rbd mirror status mirror/image_50 2024-09-11T09:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:0 2024-09-11T09:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:42.998 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:0 cluster instance 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:0 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=0 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:42.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:0 cluster1 2024-09-11T09:06:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:0 2024-09-11T09:06:43.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:0 local_cluster instance 2024-09-11T09:06:43.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:0 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=0 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 0 = cluster1:0 ']' 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=0 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=0 2024-09-11T09:06:43.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.0 'admin socket' 2024-09-11T09:06:43.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok 2024-09-11T09:06:43.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.0.asok rbd mirror status mirror/image_50 2024-09-11T09:06:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-11T09:06:43.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:43.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:1 rbd mirror status mirror/image_50 2024-09-11T09:06:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:1 2024-09-11T09:06:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:1 cluster instance 2024-09-11T09:06:43.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:1 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=1 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=1 2024-09-11T09:06:43.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:1 cluster1 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:1 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:1 local_cluster instance 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:1 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=1 2024-09-11T09:06:43.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 1 = cluster1:1 ']' 2024-09-11T09:06:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=1 2024-09-11T09:06:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=1 2024-09-11T09:06:43.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.1 'admin socket' 2024-09-11T09:06:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:43.155 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok 2024-09-11T09:06:43.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.1.asok rbd mirror status mirror/image_50 2024-09-11T09:06:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:Invalid command: unused arguments: ['mirror/image_50'] 2024-09-11T09:06:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd mirror status : get status for rbd mirror 2024-09-11T09:06:43.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:admin_socket: invalid command 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ for loop_instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ admin_daemon cluster1:2 rbd mirror status mirror/image_50 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1:2 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ shift 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ set_cluster_instance cluster1:2 cluster instance 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local val=cluster1:2 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster_var_name=cluster 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local instance_var_name=instance 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=2 2024-09-11T09:06:43.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval cluster=cluster1 2024-09-11T09:06:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ eval instance=2 2024-09-11T09:06:43.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ daemon_asok_file cluster1:2 cluster1 2024-09-11T09:06:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local local_cluster=cluster1:2 2024-09-11T09:06:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster=cluster1 2024-09-11T09:06:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance 2024-09-11T09:06:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ set_cluster_instance cluster1:2 local_cluster instance 2024-09-11T09:06:43.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local val=cluster1:2 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local cluster_var_name=local_cluster 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ local instance_var_name=instance 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ cluster=cluster1 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ instance=2 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ '[' 2 = cluster1:2 ']' 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval local_cluster=cluster1 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ local_cluster=cluster1 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ eval instance=2 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ instance=2 2024-09-11T09:06:43.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++++ ceph-conf --cluster cluster1 --name client.mirror.2 'admin socket' 2024-09-11T09:06:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+++ echo /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local asok_file=/var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ test -S /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok 2024-09-11T09:06:43.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ ceph --admin-daemon /var/run/ceph/rbd-mirror.cluster1-client.mirror.2.asok rbd mirror status mirror/image_50 2024-09-11T09:06:43.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ return 0 2024-09-11T09:06:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ status_result=' "state": "Replaying"' 2024-09-11T09:06:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo ' "state": "Replaying"' 2024-09-11T09:06:43.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ grep -i Replaying 2024-09-11T09:06:43.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: "state": "Replaying" 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=started 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test started = started 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_replay_complete cluster1 cluster2 mirror image_50 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:43.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = journal ']' 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' snapshot = snapshot ']' 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snapshot_sync_complete cluster1 cluster2 mirror image_50 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_cluster=cluster1 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster2-mirror-image_50.status 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster2-mirror-image_50.status 2024-09-11T09:06:43.439 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local status_log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.status 2024-09-11T09:06:43.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ mkfname cluster1-mirror-image_50.status 2024-09-11T09:06:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ echo cluster1-mirror-image_50.status 2024-09-11T09:06:43.441 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -e 's|[/ ]|_|g' 2024-09-11T09:06:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local local_status_log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ mirror_image_snapshot cluster2 mirror image_50 2024-09-11T09:06:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:43.443 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 mirror image snapshot mirror/image_50 2024-09-11T09:06:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Snapshot ID: 184 2024-09-11T09:06:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster2 mirror image_50 /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.status 2024-09-11T09:06:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.status 2024-09-11T09:06:44.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap list --all mirror/image_50 --format xml 2024-09-11T09:06:44.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:44.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/id 2024-09-11T09:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snapshot_id=184 2024-09-11T09:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:44.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.2 2024-09-11T09:06:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:44.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T09:06:44.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:44.345 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=84 2024-09-11T09:06:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 184 = 84 2024-09-11T09:06:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:44.348 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.4 2024-09-11T09:06:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:44.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T09:06:44.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:44.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=84 2024-09-11T09:06:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 184 = 84 2024-09-11T09:06:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:44.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.8 2024-09-11T09:06:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:45.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T09:06:45.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:45.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:06:45.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:-:1.1: Document is empty 2024-09-11T09:06:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: 2024-09-11T09:06:45.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:^ 2024-09-11T09:06:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id= 2024-09-11T09:06:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 184 = '' 2024-09-11T09:06:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.2 0.4 0.8 1.6 2 2 4 4 8 8 16 16 32 32 2024-09-11T09:06:45.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1.6 2024-09-11T09:06:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ get_newest_mirror_snapshot cluster1 mirror image_50 /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:47.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local log=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.status 2024-09-11T09:06:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 snap list --all mirror/image_50 --format xml 2024-09-11T09:06:47.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ xmlstarlet sel -t -c '//snapshots/snapshot[namespace/complete='\''true'\'' and position()=last()]' 2024-09-11T09:06:47.567 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ xmlstarlet sel -t -v //snapshot/namespace/primary_snap_id 2024-09-11T09:06:47.571 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local primary_snapshot_id=184 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 184 = 184 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_snap_present cluster1 mirror image_50 snap 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 1 2 4 8 8 8 8 8 8 8 8 16 16 16 16 32 32 32 32 2024-09-11T09:06:47.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 1 2024-09-11T09:06:48.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 info mirror/image_50@snap 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_50': 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 128 MiB in 32 objects 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 2 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 12c4742b9f79 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.12c4742b9f79 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock, non-primary 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:06:48.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 08:59:16 2024 2024-09-11T09:06:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 08:59:16 2024 2024-09-11T09:06:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 08:59:16 2024 2024-09-11T09:06:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: protected: False 2024-09-11T09:06:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:06:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:06:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 721a420b-6433-4096-afbf-22d3debada44 2024-09-11T09:06:48.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: false 2024-09-11T09:06:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:48.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ compare_image_snaps mirror image_50 snap 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap_name=snap 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local ret=0 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local rmt_export=/tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.export 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local loc_export=/tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.export 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.export 2024-09-11T09:06:48.639 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 -p mirror export image_50@snap /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.export 2024-09-11T09:06:48.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:48.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster1 -p mirror export image_50@snap /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.export 2024-09-11T09:06:49.180 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Exporting image: 3% complete... Exporting image: 6% complete... Exporting image: 9% complete... Exporting image: 12% complete... Exporting image: 15% complete... Exporting image: 18% complete... Exporting image: 21% complete... Exporting image: 25% complete... Exporting image: 28% complete... Exporting image: 31% complete... Exporting image: 34% complete... Exporting image: 37% complete... Exporting image: 40% complete... Exporting image: 43% complete... Exporting image: 46% complete... Exporting image: 50% complete... Exporting image: 53% complete... Exporting image: 56% complete... Exporting image: 59% complete... Exporting image: 62% complete... Exporting image: 65% complete... Exporting image: 68% complete... Exporting image: 71% complete... Exporting image: 75% complete... Exporting image: 78% complete... Exporting image: 81% complete... Exporting image: 84% complete... Exporting image: 87% complete... Exporting image: 90% complete... Exporting image: 93% complete... Exporting image: 96% complete... Exporting image: 100% complete...done. 2024-09-11T09:06:49.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cmp /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.export 2024-09-11T09:06:49.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -f /tmp/tmp.LFCa6yxq8J/cluster2-mirror-image_50.export /tmp/tmp.LFCa6yxq8J/cluster1-mirror-image_50.export 2024-09-11T09:06:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ testlog 'TEST: delete many images' 2024-09-11T09:06:49.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ date '+%F %T' 2024-09-11T09:06:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ tee -a /tmp/tmp.LFCa6yxq8J/rbd-mirror.test.log 2024-09-11T09:06:49.311 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo 2024-09-11 09:06:49 TEST: delete many images 2024-09-11T09:06:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:2024-09-11 09:06:49 TEST: delete many images 2024-09-11T09:06:49.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 50 2024-09-11T09:06:49.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_1 2024-09-11T09:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_1 snap 2024-09-11T09:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:06:49.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_1@snap 2024-09-11T09:06:49.672 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:06:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-11T09:06:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:06:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:06:49.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:06:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_1 2024-09-11T09:06:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:06:49.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-11T09:06:49.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-11T09:06:49.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:06:49.905+0000 7f27bc499640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62eeb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:06:49.906 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:06:49.905+0000 7f27c2ca6640 -1 rbd::mirror::ImageReplayer: 0x5581f62b0500 [3/209601d8-0ce8-4bab-bfd1-bea5e9278d31] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:06:50.890 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_2 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_2 snap 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:06:50.894 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_2@snap 2024-09-11T09:06:51.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:06:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-11T09:06:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:51.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:06:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:06:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:06:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_2 2024-09-11T09:06:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:06:51.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-11T09:06:51.779 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-11T09:06:51.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:06:51.917+0000 7f3bd016b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c09ff316c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:06:51.916 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:06:51.917+0000 7f3bd6177640 -1 rbd::mirror::ImageReplayer: 0x55c09fd53900 [3/6176f630-2f7a-4485-9979-55a0a1132ec3] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:06:52.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_3 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_3 snap 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:06:52.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_3@snap 2024-09-11T09:06:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:06:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-11T09:06:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:06:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:06:53.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:06:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_3 2024-09-11T09:06:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:06:53.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-11T09:06:53.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-11T09:06:53.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:06:53.949+0000 7f27bbc98640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62efd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:06:53.949 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:06:53.949+0000 7f27c24a5640 -1 rbd::mirror::ImageReplayer: 0x5581f63be280 [3/8efd8144-a459-4554-b275-2ec378bad1af] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:06:54.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:06:54.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_4 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_4 snap 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:06:54.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_4@snap 2024-09-11T09:06:55.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:06:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-11T09:06:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:06:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:06:55.775 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:06:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_4 2024-09-11T09:06:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:06:55.776 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-11T09:06:55.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-11T09:06:55.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:06:55.969+0000 7f3bd016b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c0a03aab60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:06:55.970 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:06:55.969+0000 7f3bd6177640 -1 rbd::mirror::ImageReplayer: 0x55c09fd52c80 [3/39633655-9b79-45e6-ac71-541b46d2ea10] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:06:56.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_5 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_5 snap 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:06:56.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_5@snap 2024-09-11T09:06:57.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:06:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-11T09:06:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:06:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:06:57.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:06:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_5 2024-09-11T09:06:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:06:57.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-11T09:06:57.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-11T09:06:57.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:06:57.981+0000 7f27bc499640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62ef1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:06:57.983 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:06:57.981+0000 7f27c2ca6640 -1 rbd::mirror::ImageReplayer: 0x5581f62b0f00 [3/57b1ae5d-d195-4c6d-b9e5-3241f71e3de5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:06:58.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:06:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:06:58.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:06:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_6 2024-09-11T09:06:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_6 snap 2024-09-11T09:06:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:06:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:06:58.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_6@snap 2024-09-11T09:06:59.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:06:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-11T09:06:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:06:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:06:59.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:06:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_6 2024-09-11T09:06:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:06:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:06:59.804 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:06:59.805 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-11T09:06:59.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-11T09:07:00.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:00.005+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece5352820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:00.006 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:00.005+0000 7fd68b1f1640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3a000 [3/bbfebdd5-c61c-4f17-9e2f-1d0a11cb2d73] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:00.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_7 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_7 snap 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:00.955 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_7@snap 2024-09-11T09:07:01.770 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-11T09:07:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:01.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:07:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:01.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_7 2024-09-11T09:07:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:07:01.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-11T09:07:01.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-11T09:07:01.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:01.997+0000 7f27bc499640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62ef6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:01.996 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:01.997+0000 7f27c2ca6640 -1 rbd::mirror::ImageReplayer: 0x5581f63be000 [3/75596a83-10f1-4277-ad91-67db500695a7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:02.961 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_8 2024-09-11T09:07:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_8 snap 2024-09-11T09:07:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:02.965 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:07:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:02.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_8@snap 2024-09-11T09:07:03.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-11T09:07:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:07:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:03.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_8 2024-09-11T09:07:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:03.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:07:03.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-11T09:07:03.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-11T09:07:03.875 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:03.878+0000 7f6578d84640 -1 reset not still connected to 0x55b98c8fcb60 2024-09-11T09:07:03.876 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:03.878+0000 7f6578d84640 -1 reset not still connected to 0x55b98d4a7e10 2024-09-11T09:07:03.878 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:03.882+0000 7f6578d84640 -1 reset not still connected to 0x55b98f4cac30 2024-09-11T09:07:03.878 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:03.882+0000 7f6578d84640 -1 reset not still connected to 0x55b98f4cb110 2024-09-11T09:07:03.878 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:03.882+0000 7f6578d84640 -1 reset not still connected to 0x55b992b0c680 2024-09-11T09:07:03.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:03.929+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece53529c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:03.929 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:03.929+0000 7fd68b1f1640 -1 rbd::mirror::ImageReplayer: 0x55ece4d32280 [3/880a8fca-463b-4b43-b6d1-4b6ea06390b7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:04.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_9 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_9 snap 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:04.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_9@snap 2024-09-11T09:07:05.083 INFO:tasks.ceph.cluster1.osd.0.smithi073.stderr:2024-09-11T09:07:05.086+0000 7f5f47055640 -1 reset not still connected to 0x55d10e9c1ad0 2024-09-11T09:07:05.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:05.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-11T09:07:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:07:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:05.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_9 2024-09-11T09:07:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:07:05.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-11T09:07:05.885 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-11T09:07:06.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:06.025+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece5147a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:06.025 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:06.025+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3b680 [3/923d86f9-42b1-48a1-b1b7-848b1ff22bf5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:06.083 INFO:tasks.ceph.cluster1.osd.0.smithi073.stderr:2024-09-11T09:07:06.082+0000 7f5f47055640 -1 reset not still connected to 0x55d10e9c1ad0 2024-09-11T09:07:07.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_10 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_10 snap 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:07.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_10@snap 2024-09-11T09:07:07.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-11T09:07:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:07:07.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_10 2024-09-11T09:07:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:07.857 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:07:07.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-11T09:07:07.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-11T09:07:08.048 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:08.049+0000 7f7d07fab640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e448e340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:08.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:08.049+0000 7f7d0e7b8640 -1 rbd::mirror::ImageReplayer: 0x55e1e41ff400 [3/b544cd07-41a2-4db4-b006-4225d4700d0a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:08.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_11 2024-09-11T09:07:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_11 snap 2024-09-11T09:07:08.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:07:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:08.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_11@snap 2024-09-11T09:07:09.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_11 2024-09-11T09:07:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:09.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:07:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:09.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_11 2024-09-11T09:07:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:07:09.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_11 2024-09-11T09:07:09.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_11 2024-09-11T09:07:10.049 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:10.049+0000 7f3bd016b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c0a03ab1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:11.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_12 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_12 snap 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:11.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_12@snap 2024-09-11T09:07:11.840 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_12 2024-09-11T09:07:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:07:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:11.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_12 2024-09-11T09:07:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:11.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:07:11.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_12 2024-09-11T09:07:11.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_12 2024-09-11T09:07:12.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:12.109+0000 7f27bc499640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62eeb60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:12.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:12.109+0000 7f27c2ca6640 -1 rbd::mirror::ImageReplayer: 0x5581f62b0280 [3/0791c4fe-f9ee-44f1-8b14-208766525577] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:13.055 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_13 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_13 snap 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:13.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_13@snap 2024-09-11T09:07:13.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:13.743 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_13 2024-09-11T09:07:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:07:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:13.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_13 2024-09-11T09:07:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:07:13.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_13 2024-09-11T09:07:13.798 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_13 2024-09-11T09:07:13.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:13.941+0000 7f27bbc98640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62ef1e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:13.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:13.941+0000 7f27c24a5640 -1 rbd::mirror::ImageReplayer: 0x5581f62b0c80 [3/551a4abb-edcd-4eb2-8d3f-d80000f55c42] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:15.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_14 2024-09-11T09:07:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_14 snap 2024-09-11T09:07:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:15.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:07:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:15.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_14@snap 2024-09-11T09:07:15.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_14 2024-09-11T09:07:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:07:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:15.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:15.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_14 2024-09-11T09:07:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:07:15.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_14 2024-09-11T09:07:15.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_14 2024-09-11T09:07:16.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:16.181+0000 7f3bcf96a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c09ff31a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:16.181 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:16.181+0000 7f3bd6978640 -1 rbd::mirror::ImageReplayer: 0x55c09fd53400 [3/4174ebf4-c3c8-4e27-a392-9c68a9fb6a29] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:17.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_15 2024-09-11T09:07:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_15 snap 2024-09-11T09:07:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:17.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:07:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:17.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_15@snap 2024-09-11T09:07:17.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:17.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_15 2024-09-11T09:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:17.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_15 2024-09-11T09:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:07:17.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_15 2024-09-11T09:07:17.960 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_15 2024-09-11T09:07:18.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:18.125+0000 7f27bc499640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62efba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:18.126 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:18.125+0000 7f27c2ca6640 -1 rbd::mirror::ImageReplayer: 0x5581f62b1900 [3/5f90212f-9abd-40ef-98b3-fab670ebcab5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:18.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:18.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_16 2024-09-11T09:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_16 snap 2024-09-11T09:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:18.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_16@snap 2024-09-11T09:07:19.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_16 2024-09-11T09:07:19.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:07:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:19.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_16 2024-09-11T09:07:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:19.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:07:19.939 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_16 2024-09-11T09:07:19.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_16 2024-09-11T09:07:20.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:20.129+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece5147ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:20.129 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:20.129+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3af00 [3/82fa71ee-6b32-459f-a37e-db8386916102] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:21.108 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:21.112 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_17 2024-09-11T09:07:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_17 snap 2024-09-11T09:07:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:07:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:21.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_17@snap 2024-09-11T09:07:21.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_17 2024-09-11T09:07:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:21.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:07:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_17 2024-09-11T09:07:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:21.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:07:21.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_17 2024-09-11T09:07:21.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_17 2024-09-11T09:07:22.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:22.157+0000 7f7d07fab640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e4201ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:22.159 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:22.157+0000 7f7d0efb9640 -1 rbd::mirror::ImageReplayer: 0x55e1e41fe500 [3/28968644-b7f0-40d0-b9ce-2784d7675298] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:23.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_18 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_18 snap 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:23.114 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_18@snap 2024-09-11T09:07:23.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_18 2024-09-11T09:07:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:07:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_18 2024-09-11T09:07:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:07:23.745 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_18 2024-09-11T09:07:23.800 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_18 2024-09-11T09:07:24.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:24.169+0000 7f27bbc98640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f5da2340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:24.169 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:24.169+0000 7f27c24a5640 -1 rbd::mirror::ImageReplayer: 0x5581f62b0a00 [3/350bf260-3a2c-48eb-8813-d7dafe84bdb1] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:25.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:25.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_19 2024-09-11T09:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_19 snap 2024-09-11T09:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:25.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_19@snap 2024-09-11T09:07:25.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:25.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_19 2024-09-11T09:07:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:07:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:25.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_19 2024-09-11T09:07:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:07:25.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_19 2024-09-11T09:07:26.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_19 2024-09-11T09:07:26.157 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:26.158+0000 7f6578d84640 -1 reset not still connected to 0x55b993b11450 2024-09-11T09:07:26.158 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:26.158+0000 7f6578d84640 -1 reset not still connected to 0x55b993b116c0 2024-09-11T09:07:26.158 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:26.158+0000 7f6578d84640 -1 reset not still connected to 0x55b99489eb60 2024-09-11T09:07:26.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:26.193+0000 7f27bbc98640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62ee820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:26.195 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:26.193+0000 7f27c24a5640 -1 rbd::mirror::ImageReplayer: 0x5581f62b1180 [3/05c8607a-0bf6-407e-b458-e04320b3a7ac] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:27.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_20 2024-09-11T09:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_20 snap 2024-09-11T09:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:27.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:07:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:27.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_20@snap 2024-09-11T09:07:27.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_20 2024-09-11T09:07:27.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:07:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:27.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_20 2024-09-11T09:07:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:07:27.982 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_20 2024-09-11T09:07:28.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_20 2024-09-11T09:07:28.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:28.224+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece5147d40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:28.226 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:28.224+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3b400 [3/6bae77b4-74ad-4d05-87e9-78bd958ad83a] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:28.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_21 2024-09-11T09:07:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_21 snap 2024-09-11T09:07:28.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:07:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:28.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_21@snap 2024-09-11T09:07:29.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:30.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_21 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_21 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:07:30.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_21 2024-09-11T09:07:30.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_21 2024-09-11T09:07:30.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:30.216+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4fca820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:30.219 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:30.216+0000 7fd68b1f1640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3b900 [3/ae0c9716-d47a-4a34-8cf9-e01df79ded5c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:30.317 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7635765783961403) > weight (0.25) 2024-09-11T09:07:30.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.957995718646758) > weight (0.25) 2024-09-11T09:07:30.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.5629142079198072) > weight (0.25) 2024-09-11T09:07:30.318 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4069275729018257) > weight (0.25) 2024-09-11T09:07:30.318 INFO:tasks.rbd_mirror_thrash:waiting for 71 secs before thrashing 2024-09-11T09:07:31.173 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:31.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_22 2024-09-11T09:07:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_22 snap 2024-09-11T09:07:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:07:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:31.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_22@snap 2024-09-11T09:07:31.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_22 2024-09-11T09:07:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:07:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:32.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_22 2024-09-11T09:07:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:07:32.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_22 2024-09-11T09:07:32.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_22 2024-09-11T09:07:32.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:32.260+0000 7f3bd016b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c09ff301a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:32.260 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:32.260+0000 7f3bd6978640 -1 rbd::mirror::ImageReplayer: 0x55c09fd52500 [3/9ceefcdb-b3e9-4601-8f51-a204c212530d] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:33.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:33.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_23 2024-09-11T09:07:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_23 snap 2024-09-11T09:07:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:07:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:33.199 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_23@snap 2024-09-11T09:07:33.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_23 2024-09-11T09:07:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:33.738 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:07:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_23 2024-09-11T09:07:33.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:07:33.740 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_23 2024-09-11T09:07:33.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_23 2024-09-11T09:07:33.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:33.932+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece5147ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:33.933 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:33.932+0000 7fd68b1f1640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3b180 [3/69e0ab8a-356e-4e12-8363-b402fa05e07f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:34.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:34.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_24 2024-09-11T09:07:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_24 snap 2024-09-11T09:07:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:07:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:34.898 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_24@snap 2024-09-11T09:07:35.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_24 2024-09-11T09:07:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:35.755 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:07:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_24 2024-09-11T09:07:35.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:07:35.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_24 2024-09-11T09:07:35.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_24 2024-09-11T09:07:35.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:35.960+0000 7f3bcf96a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c0a03aad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:35.960 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:35.960+0000 7f3bd6177640 -1 rbd::mirror::ImageReplayer: 0x55c09fd53680 [3/523e3106-9d30-48a3-8d7d-0a45f9581e5e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:36.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_25 2024-09-11T09:07:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_25 snap 2024-09-11T09:07:36.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:07:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:36.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_25@snap 2024-09-11T09:07:37.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_25 2024-09-11T09:07:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:07:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:37.763 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_25 2024-09-11T09:07:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:07:37.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_25 2024-09-11T09:07:37.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_25 2024-09-11T09:07:37.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:37.964+0000 7f7d07fab640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e4201a00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:37.965 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:37.964+0000 7f7d0efb9640 -1 rbd::mirror::ImageReplayer: 0x55e1e41fe000 [3/0534cedd-e737-4e3d-87e8-aff6ea511c46] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:38.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_26 2024-09-11T09:07:38.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_26 snap 2024-09-11T09:07:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:07:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:38.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_26@snap 2024-09-11T09:07:39.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_26 2024-09-11T09:07:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:07:39.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:39.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_26 2024-09-11T09:07:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:07:39.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_26 2024-09-11T09:07:39.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_26 2024-09-11T09:07:39.992 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:39.992+0000 7f7d07fab640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e42011e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:39.993 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:39.992+0000 7f7d0efb9640 -1 rbd::mirror::ImageReplayer: 0x55e1e41ffb80 [3/de26d284-def3-43fc-8308-3c61d65c9c5f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:40.020 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:40.021+0000 7f6578d84640 -1 reset not still connected to 0x55b98cd35450 2024-09-11T09:07:40.020 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:40.021+0000 7f6578d84640 -1 reset not still connected to 0x55b98d4a7e10 2024-09-11T09:07:40.020 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:40.021+0000 7f6578d84640 -1 reset not still connected to 0x55b98e9dd790 2024-09-11T09:07:40.028 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:40.025+0000 7f6578d84640 -1 reset not still connected to 0x55b98f4cac30 2024-09-11T09:07:40.028 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:40.025+0000 7f6578d84640 -1 reset not still connected to 0x55b98f4cb110 2024-09-11T09:07:40.028 INFO:tasks.ceph.cluster1.osd.2.smithi073.stderr:2024-09-11T09:07:40.025+0000 7f6578d84640 -1 reset not still connected to 0x55b992b0c680 2024-09-11T09:07:40.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_27 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_27 snap 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:07:40.946 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:40.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_27@snap 2024-09-11T09:07:41.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:41.790 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_27 2024-09-11T09:07:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:07:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:41.791 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_27 2024-09-11T09:07:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:07:41.792 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_27 2024-09-11T09:07:41.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_27 2024-09-11T09:07:41.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:41.988+0000 7f7d07fab640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e448e4e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:41.990 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:41.988+0000 7f7d0e7b8640 -1 rbd::mirror::ImageReplayer: 0x55e1e41fef00 [3/8c3ff342-3906-447a-9223-131a82f747f2] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:42.412 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.3286149214075139) > weight (0.25) 2024-09-11T09:07:42.412 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.6038623601859241) > weight (0.25) 2024-09-11T09:07:42.412 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T09:07:42.413 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T09:07:42.413 INFO:tasks.rbd_mirror_thrash:waiting for 1 secs before reviving daemons 2024-09-11T09:07:42.414 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:07:42.412+0000 7fc029542640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 40809) UID: 1000 2024-09-11T09:07:42.962 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:42.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_28 2024-09-11T09:07:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_28 snap 2024-09-11T09:07:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:07:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:42.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_28@snap 2024-09-11T09:07:43.414 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T09:07:43.414 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:07:43.415 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T09:07:43.415 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T09:07:43.415 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T09:07:43.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T09:07:43.418 INFO:tasks.rbd_mirror_thrash:waiting for 65 secs before thrashing 2024-09-11T09:07:43.701 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_28 2024-09-11T09:07:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:07:43.714 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_28 2024-09-11T09:07:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:43.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:07:43.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_28 2024-09-11T09:07:43.774 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_28 2024-09-11T09:07:43.840 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:07:43.840+0000 7febc8f34640 -1 rbd::mirror::ImageMap: 0x560f3bb99c20 handle_peer_ack_remove: failed to remove global_image_id=fece848e-5f36-407c-83b5-17909bfc414d 2024-09-11T09:07:44.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:44.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_29 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_29 snap 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:44.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_29@snap 2024-09-11T09:07:45.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_29 2024-09-11T09:07:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:07:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:45.846 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_29 2024-09-11T09:07:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:07:45.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_29 2024-09-11T09:07:45.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_29 2024-09-11T09:07:46.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:46.040+0000 7f27bbc98640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62eed00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:46.040 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:46.040+0000 7f27c24a5640 -1 rbd::mirror::ImageReplayer: 0x5581f62b0780 [3/2a587c46-253d-496d-be6f-0d9d4063df36] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:47.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_30 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_30 snap 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:47.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_30@snap 2024-09-11T09:07:47.797 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_30 2024-09-11T09:07:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:07:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:47.818 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_30 2024-09-11T09:07:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:07:47.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_30 2024-09-11T09:07:47.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_30 2024-09-11T09:07:48.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:48.264+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4fcb6c0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:48.264 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:07:48.264+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3a780 [3/eb74963f-2883-4117-8704-7dd16bb13e33] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:48.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:48.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_31 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_31 snap 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:48.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_31@snap 2024-09-11T09:07:49.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_31 2024-09-11T09:07:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:07:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:49.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_31 2024-09-11T09:07:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:07:49.850 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_31 2024-09-11T09:07:49.904 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_31 2024-09-11T09:07:50.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:50.120+0000 7f7d07fab640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e4201860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:50.119 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:50.120+0000 7f7d0efb9640 -1 rbd::mirror::ImageReplayer: 0x55e1e41ff180 [3/a8e3428e-5e9d-4305-a12d-e6118689e825] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:51.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_32 2024-09-11T09:07:51.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_32 snap 2024-09-11T09:07:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:07:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:51.020 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_32@snap 2024-09-11T09:07:51.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_32 2024-09-11T09:07:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:07:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:51.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_32 2024-09-11T09:07:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:07:51.835 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_32 2024-09-11T09:07:51.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_32 2024-09-11T09:07:52.099 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:52.104+0000 7f27bc499640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62ee000 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:52.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:07:52.104+0000 7f27c24a5640 -1 rbd::mirror::ImageReplayer: 0x5581f62b1b80 [3/62829279-46f0-42ff-8ae0-7092ec6a74d0] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:53.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_33 2024-09-11T09:07:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_33 snap 2024-09-11T09:07:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:53.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:07:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:53.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_33@snap 2024-09-11T09:07:53.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_33 2024-09-11T09:07:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:07:53.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_33 2024-09-11T09:07:53.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:07:53.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_33 2024-09-11T09:07:53.802 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_33 2024-09-11T09:07:53.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:53.948+0000 7f3bcf96a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c0a03aa820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:53.945 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:07:53.948+0000 7f3bd6177640 -1 rbd::mirror::ImageReplayer: 0x55c09fd53180 [3/3b06c806-6a3f-42c2-bb4e-e1f16e05e5d7] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:55.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_34 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_34 snap 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:55.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_34@snap 2024-09-11T09:07:55.849 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_34 2024-09-11T09:07:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:07:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:55.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_34 2024-09-11T09:07:55.876 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:07:55.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_34 2024-09-11T09:07:55.947 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_34 2024-09-11T09:07:56.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:56.136+0000 7f7d087ac640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e448e680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:07:56.133 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:07:56.136+0000 7f7d0e7b8640 -1 rbd::mirror::ImageReplayer: 0x55e1e41fec80 [3/75f0a352-e3d0-46c7-8e9c-cd21648d227e] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:07:57.043 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_35 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_35 snap 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:57.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_35@snap 2024-09-11T09:07:57.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_35 2024-09-11T09:07:57.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:07:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:57.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_35 2024-09-11T09:07:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:07:57.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_35 2024-09-11T09:07:57.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_35 2024-09-11T09:07:58.007 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:07:58.012+0000 7febc9735640 -1 rbd::mirror::ImageMap: 0x560f3bb99c20 handle_peer_ack_remove: failed to remove global_image_id=d2627e7d-0b6b-467d-bdbf-be0af065f5f6 2024-09-11T09:07:58.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:07:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:07:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:07:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_36 2024-09-11T09:07:58.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_36 snap 2024-09-11T09:07:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:07:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:07:58.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_36@snap 2024-09-11T09:07:59.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:07:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_36 2024-09-11T09:07:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:59.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:07:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:07:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:07:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_36 2024-09-11T09:07:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:07:59.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:07:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:07:59.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_36 2024-09-11T09:07:59.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_36 2024-09-11T09:08:00.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:00.112+0000 7f3bcf96a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c09ff31520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:00.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:00.112+0000 7f3bd6978640 -1 rbd::mirror::ImageReplayer: 0x55c09fe26f00 [3/f6933439-7dab-4216-b4b5-b329ed22c0f6] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:01.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:01.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_37 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_37 snap 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:01.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_37@snap 2024-09-11T09:08:01.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_37 2024-09-11T09:08:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:01.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:08:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:01.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_37 2024-09-11T09:08:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:08:01.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_37 2024-09-11T09:08:01.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_37 2024-09-11T09:08:02.041 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:08:02.044+0000 7febc9735640 -1 rbd::mirror::ImageMap: 0x560f3bb99c20 handle_peer_ack_remove: failed to remove global_image_id=a953381e-b730-4277-a246-b871eee2718e 2024-09-11T09:08:03.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_38 2024-09-11T09:08:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_38 snap 2024-09-11T09:08:03.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:08:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:03.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_38@snap 2024-09-11T09:08:03.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_38 2024-09-11T09:08:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:08:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:03.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_38 2024-09-11T09:08:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:08:03.758 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_38 2024-09-11T09:08:03.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_38 2024-09-11T09:08:03.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:08:03.964+0000 7f7d087ac640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e448e820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:03.962 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:08:03.964+0000 7f7d0e7b8640 -1 rbd::mirror::ImageReplayer: 0x55e1e42e0000 [3/e269182c-689f-4c97-bd46-05795a725cb4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:05.178 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_39 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_39 snap 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:05.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_39@snap 2024-09-11T09:08:05.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_39 2024-09-11T09:08:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:08:05.941 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:05.942 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_39 2024-09-11T09:08:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:08:05.943 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_39 2024-09-11T09:08:06.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_39 2024-09-11T09:08:06.075 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:08:06.080+0000 7febc8f34640 -1 rbd::mirror::ImageMap: 0x560f3bb99c20 handle_peer_ack_remove: failed to remove global_image_id=c046011c-40fa-4138-bba6-2fc652d532ec 2024-09-11T09:08:07.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_40 2024-09-11T09:08:07.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_40 snap 2024-09-11T09:08:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:08:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:07.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_40@snap 2024-09-11T09:08:07.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_40 2024-09-11T09:08:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:08:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:07.957 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_40 2024-09-11T09:08:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:08:07.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_40 2024-09-11T09:08:08.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_40 2024-09-11T09:08:08.098 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:08:08.100+0000 7febc9735640 -1 rbd::mirror::ImageMap: 0x560f3bb99c20 handle_peer_ack_remove: failed to remove global_image_id=24872562-d2cb-4569-88af-e8992d096531 2024-09-11T09:08:08.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_41 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_41 snap 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:08.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_41@snap 2024-09-11T09:08:09.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_41 2024-09-11T09:08:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:08:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:09.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_41 2024-09-11T09:08:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:09.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:08:09.971 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_41 2024-09-11T09:08:10.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_41 2024-09-11T09:08:10.193 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:08:10.196+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4979040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:10.194 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:08:10.196+0000 7fd68b1f1640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3aa00 [3/e18f305c-3600-4a05-ac01-cbd0ec5179dd] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:11.099 INFO:tasks.ceph.cluster2.osd.0.smithi077.stderr:2024-09-11T09:08:11.104+0000 7f2b03c68640 -1 reset not still connected to 0x55c6132c5e10 2024-09-11T09:08:11.161 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_42 2024-09-11T09:08:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_42 snap 2024-09-11T09:08:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:11.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:08:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:11.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_42@snap 2024-09-11T09:08:11.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_42 2024-09-11T09:08:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:08:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:12.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_42 2024-09-11T09:08:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:08:12.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_42 2024-09-11T09:08:12.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_42 2024-09-11T09:08:12.427 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:08:12.432+0000 7f27bc499640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x5581f62ee1a0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:12.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:08:12.432+0000 7f27c2ca6640 -1 rbd::mirror::ImageReplayer: 0x5581f62b1680 [3/52286b49-eaf5-465a-988f-e2ac13d2e71c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:13.186 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_43 2024-09-11T09:08:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_43 snap 2024-09-11T09:08:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:13.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:08:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:13.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_43@snap 2024-09-11T09:08:13.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_43 2024-09-11T09:08:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:08:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:13.754 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_43 2024-09-11T09:08:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:08:13.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_43 2024-09-11T09:08:13.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_43 2024-09-11T09:08:13.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:08:13.960+0000 7f7d087ac640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e3f6f860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:13.957 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:08:13.960+0000 7f7d0efb9640 -1 rbd::mirror::ImageReplayer: 0x55e1e41fea00 [3/540fcbee-d8ed-438b-8b66-b3b65770db9f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:14.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:14.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_44 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_44 snap 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_44@snap 2024-09-11T09:08:15.983 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:16.005 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_44 2024-09-11T09:08:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:08:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:16.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_44 2024-09-11T09:08:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:08:16.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_44 2024-09-11T09:08:16.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_44 2024-09-11T09:08:16.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:08:16.264+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4fcb520 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:16.263 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:08:16.264+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4d32000 [3/9eba4b2e-6f4b-4273-9365-f3717576509c] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:17.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_45 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_45 snap 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:17.192 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_45@snap 2024-09-11T09:08:17.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_45 2024-09-11T09:08:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:08:18.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:18.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_45 2024-09-11T09:08:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:08:18.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_45 2024-09-11T09:08:18.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_45 2024-09-11T09:08:18.275 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:08:18.280+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4978b60 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:18.276 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:08:18.280+0000 7fd68b1f1640 -1 rbd::mirror::ImageReplayer: 0x55ece4d32780 [3/f5150c59-0ae1-4502-a239-7a0369bc62f5] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:18.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_46 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_46 snap 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:18.918 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_46@snap 2024-09-11T09:08:20.006 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_46 2024-09-11T09:08:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:08:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:20.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_46 2024-09-11T09:08:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:20.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:08:20.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_46 2024-09-11T09:08:20.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_46 2024-09-11T09:08:20.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:20.212+0000 7f3bcf96a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c09ff304e0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:20.208 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:20.212+0000 7f3bd6177640 -1 rbd::mirror::ImageReplayer: 0x55c0a04fc000 [3/8cd87261-11aa-45a3-8d2c-211e06dbeb62] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:21.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:21.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_47 2024-09-11T09:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_47 snap 2024-09-11T09:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:21.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_47@snap 2024-09-11T09:08:22.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:22.050 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_47 2024-09-11T09:08:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:08:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:22.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_47 2024-09-11T09:08:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:08:22.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_47 2024-09-11T09:08:22.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_47 2024-09-11T09:08:22.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:22.227+0000 7f3bcf96a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c09fd8bba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:22.223 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:22.227+0000 7f3bd6177640 -1 rbd::mirror::ImageReplayer: 0x55c09fd53b80 [3/8ad010f4-22cf-4eef-8e81-0a6236a13192] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:23.214 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_48 2024-09-11T09:08:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_48 snap 2024-09-11T09:08:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:08:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:23.219 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_48@snap 2024-09-11T09:08:23.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_48 2024-09-11T09:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:23.748 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_48 2024-09-11T09:08:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:08:23.750 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_48 2024-09-11T09:08:23.806 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_48 2024-09-11T09:08:23.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:08:23.967+0000 7f7d087ac640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55e1e4171040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:23.964 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:08:23.967+0000 7f7d0efb9640 -1 rbd::mirror::ImageReplayer: 0x55e1e42e0280 [3/ef84862e-99c3-4940-9190-5144ad96e521] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:24.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:24.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_49 2024-09-11T09:08:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_49 snap 2024-09-11T09:08:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:08:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:24.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_49@snap 2024-09-11T09:08:25.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_49 2024-09-11T09:08:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:08:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:25.771 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:25.772 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_49 2024-09-11T09:08:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:08:25.773 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_49 2024-09-11T09:08:25.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_49 2024-09-11T09:08:25.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:25.979+0000 7f3bcf96a640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c0a03aa820 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:25.978 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:25.979+0000 7f3bd6978640 -1 rbd::mirror::ImageReplayer: 0x55c09fe26280 [3/ef6d8f68-5e32-4505-b07d-5119f8fb90c8] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:26.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_50 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_snapshot cluster2 mirror image_50 snap 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local snap=snap 2024-09-11T09:08:26.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap rm mirror/image_50@snap 2024-09-11T09:08:27.757 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing snap: 100% complete...done. 2024-09-11T09:08:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_50 2024-09-11T09:08:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:08:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:27.787 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_50 2024-09-11T09:08:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:08:27.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_50 2024-09-11T09:08:27.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_50 2024-09-11T09:08:27.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:27.987+0000 7f3bd016b640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55c0a03aad00 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:08:27.987 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:08:27.987+0000 7f3bd6177640 -1 rbd::mirror::ImageReplayer: 0x55c09fd52f00 [3/721a420b-6433-4096-afbf-22d3debada44] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:08:28.928 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 3% complete... Removing image: 6% complete... Removing image: 9% complete... Removing image: 12% complete... Removing image: 15% complete... Removing image: 18% complete... Removing image: 21% complete... Removing image: 25% complete... Removing image: 28% complete... Removing image: 31% complete... Removing image: 34% complete... Removing image: 37% complete... Removing image: 40% complete... Removing image: 43% complete... Removing image: 46% complete... Removing image: 50% complete... Removing image: 53% complete... Removing image: 56% complete... Removing image: 59% complete... Removing image: 62% complete... Removing image: 65% complete... Removing image: 68% complete... Removing image: 71% complete... Removing image: 75% complete... Removing image: 78% complete... Removing image: 81% complete... Removing image: 84% complete... Removing image: 87% complete... Removing image: 90% complete... Removing image: 93% complete... Removing image: 96% complete... Removing image: 100% complete...done. 2024-09-11T09:08:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:28.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ testlog 'TEST: image deletions should propagate' 2024-09-11T09:08:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ tee -a /tmp/tmp.LFCa6yxq8J/rbd-mirror.test.log 2024-09-11T09:08:28.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ date '+%F %T' 2024-09-11T09:08:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo 2024-09-11 09:08:28 TEST: image deletions should propagate 2024-09-11T09:08:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:2024-09-11 09:08:28 TEST: image deletions should propagate 2024-09-11T09:08:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_pool_images cluster1 mirror 0 2024-09-11T09:08:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_count=0 2024-09-11T09:08:28.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local count 2024-09-11T09:08:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local last_count=0 2024-09-11T09:08:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ true 2024-09-11T09:08:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 40 2024-09-11T09:08:28.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in `seq 1 40` 2024-09-11T09:08:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 1 -ne 1 2024-09-11T09:08:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T09:08:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep 'images: ' 2024-09-11T09:08:28.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cut '-d ' -f 2 2024-09-11T09:08:28.975 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T09:08:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ count=0 2024-09-11T09:08:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 = 0 2024-09-11T09:08:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:28.978 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_pool_healthy cluster1 mirror 0 2024-09-11T09:08:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state 2024-09-11T09:08:28.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 40 2024-09-11T09:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in `seq 1 40` 2024-09-11T09:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 1 -ne 1 2024-09-11T09:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster cluster1 -p mirror mirror pool status 2024-09-11T09:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep 'image health:' 2024-09-11T09:08:28.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cut '-d ' -f 3 2024-09-11T09:08:29.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: failed to query services: (13) Permission denied 2024-09-11T09:08:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ state=OK 2024-09-11T09:08:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test OK = ERROR 2024-09-11T09:08:29.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test OK = OK 2024-09-11T09:08:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:29.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 50 2024-09-11T09:08:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_1 2024-09-11T09:08:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-11T09:08:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_1 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T09:08:29.095 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T09:08:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:29.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-11T09:08:29.239 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:29.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T09:08:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_1 2024-09-11T09:08:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T09:08:29.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.284 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_2 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-11T09:08:29.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_2 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T09:08:29.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.327 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T09:08:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:29.331 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:29.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-11T09:08:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:08:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:29.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T09:08:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_2 2024-09-11T09:08:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T09:08:29.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T09:08:29.483 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_3 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T09:08:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_3 2024-09-11T09:08:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T09:08:29.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T09:08:29.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:29.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-11T09:08:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:08:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:29.632 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:29.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T09:08:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_3 2024-09-11T09:08:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T09:08:29.634 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.681 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T09:08:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:29.685 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_4 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:29.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_4 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T09:08:29.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T09:08:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-11T09:08:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:08:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:29.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T09:08:29.831 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_4 2024-09-11T09:08:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T09:08:29.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_5 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-11T09:08:29.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_5 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T09:08:29.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:29.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T09:08:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:29.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-11T09:08:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.027 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:08:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:30.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T09:08:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_5 2024-09-11T09:08:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T09:08:30.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.281 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T09:08:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:30.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_6 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T09:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_6 2024-09-11T09:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T09:08:30.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T09:08:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:30.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:30.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T09:08:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_6 2024-09-11T09:08:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T09:08:30.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T09:08:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:30.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_7 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_7 2024-09-11T09:08:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T09:08:30.487 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T09:08:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:30.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-11T09:08:30.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:08:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:30.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T09:08:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_7 2024-09-11T09:08:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T09:08:30.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_8 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:08:30.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_8 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T09:08:30.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.717 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T09:08:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:30.721 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:30.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-11T09:08:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:08:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:30.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T09:08:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_8 2024-09-11T09:08:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T09:08:30.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T09:08:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:30.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_9 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:30.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_9 2024-09-11T09:08:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T09:08:30.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:30.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T09:08:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:30.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:31.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-11T09:08:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:08:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:31.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T09:08:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_9 2024-09-11T09:08:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T09:08:31.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T09:08:31.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_10 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T09:08:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_10 2024-09-11T09:08:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T09:08:31.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.115 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T09:08:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:31.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:31.221 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-11T09:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:31.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T09:08:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_10 2024-09-11T09:08:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T09:08:31.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.269 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T09:08:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:31.274 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_11 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_11 deleted 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_11 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.275 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_11 2024-09-11T09:08:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-11T09:08:31.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.324 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-11T09:08:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:31.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_11 deleted '' 2024-09-11T09:08:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.429 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_11 2024-09-11T09:08:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:31.430 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_11 2024-09-11T09:08:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_11 2024-09-11T09:08:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_11 2024-09-11T09:08:31.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.476 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_11: (2) No such file or directory 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_12 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_12 deleted 2024-09-11T09:08:31.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_12 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_12 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-11T09:08:31.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-11T09:08:31.524 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:31.525 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_12 deleted '' 2024-09-11T09:08:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_12 2024-09-11T09:08:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:31.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_12 2024-09-11T09:08:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_12 2024-09-11T09:08:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_12 2024-09-11T09:08:31.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.673 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_12: (2) No such file or directory 2024-09-11T09:08:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_13 2024-09-11T09:08:31.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_13 deleted 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_13 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_13 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-11T09:08:31.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-11T09:08:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:31.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_13 deleted '' 2024-09-11T09:08:31.820 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_13 2024-09-11T09:08:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:31.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_13 2024-09-11T09:08:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_13 2024-09-11T09:08:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_13 2024-09-11T09:08:31.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_13: (2) No such file or directory 2024-09-11T09:08:31.870 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_14 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_14 deleted 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_14 2024-09-11T09:08:31.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_14 2024-09-11T09:08:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-11T09:08:31.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:31.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-11T09:08:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:31.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_14 deleted '' 2024-09-11T09:08:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_14 2024-09-11T09:08:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:32.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_14 2024-09-11T09:08:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_14 2024-09-11T09:08:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_14 2024-09-11T09:08:32.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_14: (2) No such file or directory 2024-09-11T09:08:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:32.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_15 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_15 deleted 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_15 2024-09-11T09:08:32.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_15 2024-09-11T09:08:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-11T09:08:32.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-11T09:08:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:32.138 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:32.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_15 deleted '' 2024-09-11T09:08:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_15 2024-09-11T09:08:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:32.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_15 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_15 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_15 2024-09-11T09:08:32.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_15: (2) No such file or directory 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_16 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_16 deleted 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_16 2024-09-11T09:08:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_16 2024-09-11T09:08:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-11T09:08:32.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.328 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-11T09:08:32.332 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:32.333 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_16 deleted '' 2024-09-11T09:08:32.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_16 2024-09-11T09:08:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:32.435 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_16 2024-09-11T09:08:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_16 2024-09-11T09:08:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_16 2024-09-11T09:08:32.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.479 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_16: (2) No such file or directory 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_17 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_17 deleted 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_17 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_17 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-11T09:08:32.486 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-11T09:08:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:32.527 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_17 deleted '' 2024-09-11T09:08:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_17 2024-09-11T09:08:32.628 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:32.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_17 2024-09-11T09:08:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_17 2024-09-11T09:08:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_17 2024-09-11T09:08:32.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_17: (2) No such file or directory 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_18 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_18 deleted 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:08:32.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_18 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_18 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-11T09:08:32.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-11T09:08:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:32.722 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_18 deleted '' 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_18 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:32.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_18 2024-09-11T09:08:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_18 2024-09-11T09:08:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_18 2024-09-11T09:08:32.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_18: (2) No such file or directory 2024-09-11T09:08:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:32.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_19 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_19 deleted 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_19 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:32.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_19 2024-09-11T09:08:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-11T09:08:32.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:32.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-11T09:08:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:32.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_19 deleted '' 2024-09-11T09:08:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_19 2024-09-11T09:08:33.023 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:33.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_19 2024-09-11T09:08:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_19 2024-09-11T09:08:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_19 2024-09-11T09:08:33.025 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_19: (2) No such file or directory 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_20 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_20 deleted 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:08:33.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_20 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_20 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-11T09:08:33.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-11T09:08:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:33.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_20 deleted '' 2024-09-11T09:08:33.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_20 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_20 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_20 2024-09-11T09:08:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_20 2024-09-11T09:08:33.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_20: (2) No such file or directory 2024-09-11T09:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_21 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_21 deleted 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:33.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_21 2024-09-11T09:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_21 2024-09-11T09:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-11T09:08:33.289 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-11T09:08:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:33.329 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_21 deleted '' 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_21 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:33.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_21 2024-09-11T09:08:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_21 2024-09-11T09:08:33.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_21 2024-09-11T09:08:33.433 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_21: (2) No such file or directory 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_22 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_22 deleted 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:08:33.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_22 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_22 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-11T09:08:33.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.520 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-11T09:08:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:33.524 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_22 deleted '' 2024-09-11T09:08:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_22 2024-09-11T09:08:33.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_22 2024-09-11T09:08:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_22 2024-09-11T09:08:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_22 2024-09-11T09:08:33.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_22: (2) No such file or directory 2024-09-11T09:08:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:33.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_23 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_23 deleted 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:33.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_23 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_23 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-11T09:08:33.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-11T09:08:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:33.723 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_23 deleted '' 2024-09-11T09:08:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_23 2024-09-11T09:08:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:33.824 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_23 2024-09-11T09:08:33.825 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_23 2024-09-11T09:08:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_23 2024-09-11T09:08:33.826 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:33.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_23: (2) No such file or directory 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_24 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_24 deleted 2024-09-11T09:08:33.879 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_24 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:33.880 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_24 2024-09-11T09:08:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-11T09:08:33.881 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:34.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-11T09:08:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:34.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_24 deleted '' 2024-09-11T09:08:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_24 2024-09-11T09:08:34.230 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_24 2024-09-11T09:08:34.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_24 2024-09-11T09:08:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_24 2024-09-11T09:08:34.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:34.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_24: (2) No such file or directory 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_25 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_25 deleted 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:34.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_25 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_25 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-11T09:08:34.283 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:34.321 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-11T09:08:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:34.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:34.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_25 deleted '' 2024-09-11T09:08:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_25 2024-09-11T09:08:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:34.427 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_25 2024-09-11T09:08:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_25 2024-09-11T09:08:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:34.428 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_25 2024-09-11T09:08:34.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_25: (2) No such file or directory 2024-09-11T09:08:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:34.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_26 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_26 deleted 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_26 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_26 2024-09-11T09:08:34.481 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-11T09:08:34.482 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:34.518 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-11T09:08:34.521 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:34.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_26 deleted '' 2024-09-11T09:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:34.623 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_26 2024-09-11T09:08:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:34.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_26 2024-09-11T09:08:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_26 2024-09-11T09:08:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_26 2024-09-11T09:08:34.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:34.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_26: (2) No such file or directory 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_27 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_27 deleted 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:34.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_27 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_27 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-11T09:08:34.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:34.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-11T09:08:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:34.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_27 deleted '' 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_27 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:35.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_27 2024-09-11T09:08:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_27 2024-09-11T09:08:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_27 2024-09-11T09:08:35.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_27: (2) No such file or directory 2024-09-11T09:08:35.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_28 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_28 deleted 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_28 2024-09-11T09:08:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_28 2024-09-11T09:08:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-11T09:08:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-11T09:08:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:35.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_28 deleted '' 2024-09-11T09:08:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_28 2024-09-11T09:08:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:35.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_28 2024-09-11T09:08:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_28 2024-09-11T09:08:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_28 2024-09-11T09:08:35.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_28: (2) No such file or directory 2024-09-11T09:08:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:35.300 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_29 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_29 deleted 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_29 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.301 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_29 2024-09-11T09:08:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-11T09:08:35.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-11T09:08:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:35.342 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:35.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_29 deleted '' 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_29 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_29 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_29 2024-09-11T09:08:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_29 2024-09-11T09:08:35.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_29: (2) No such file or directory 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_30 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_30 deleted 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_30 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_30 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-11T09:08:35.496 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-11T09:08:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:35.538 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_30 deleted '' 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_30 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:35.640 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_30 2024-09-11T09:08:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_30 2024-09-11T09:08:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_30 2024-09-11T09:08:35.641 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.686 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_30: (2) No such file or directory 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_31 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_31 deleted 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:35.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_31 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_31 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-11T09:08:35.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-11T09:08:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:35.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_31 deleted '' 2024-09-11T09:08:35.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_31 2024-09-11T09:08:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:35.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_31 2024-09-11T09:08:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_31 2024-09-11T09:08:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_31 2024-09-11T09:08:35.838 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.882 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_31: (2) No such file or directory 2024-09-11T09:08:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:35.887 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_32 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_32 deleted 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_32 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:35.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_32 2024-09-11T09:08:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-11T09:08:35.889 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:35.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-11T09:08:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:35.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:36.031 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_32 deleted '' 2024-09-11T09:08:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_32 2024-09-11T09:08:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:36.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_32 2024-09-11T09:08:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_32 2024-09-11T09:08:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_32 2024-09-11T09:08:36.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_32: (2) No such file or directory 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_33 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_33 deleted 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:36.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_33 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_33 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-11T09:08:36.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-11T09:08:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:36.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:36.238 INFO:tasks.ceph.cluster1.osd.0.smithi073.stderr:2024-09-11T09:08:36.240+0000 7f5f47055640 -1 reset not still connected to 0x55d10c352340 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_33 deleted '' 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_33 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:36.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_33 2024-09-11T09:08:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_33 2024-09-11T09:08:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_33 2024-09-11T09:08:36.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_33: (2) No such file or directory 2024-09-11T09:08:36.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_34 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_34 deleted 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:36.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_34 2024-09-11T09:08:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_34 2024-09-11T09:08:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-11T09:08:36.295 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-11T09:08:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:36.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:36.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_34 deleted '' 2024-09-11T09:08:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_34 2024-09-11T09:08:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:36.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_34 2024-09-11T09:08:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_34 2024-09-11T09:08:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_34 2024-09-11T09:08:36.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.484 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_34: (2) No such file or directory 2024-09-11T09:08:36.489 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_35 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_35 deleted 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_35 2024-09-11T09:08:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_35 2024-09-11T09:08:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-11T09:08:36.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.531 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-11T09:08:36.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:36.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_35 deleted '' 2024-09-11T09:08:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_35 2024-09-11T09:08:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.636 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_35 2024-09-11T09:08:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.637 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_35 2024-09-11T09:08:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_35 2024-09-11T09:08:36.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.682 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_35: (2) No such file or directory 2024-09-11T09:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_36 2024-09-11T09:08:36.687 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_36 deleted 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_36 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_36 2024-09-11T09:08:36.688 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-11T09:08:36.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.727 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-11T09:08:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:36.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_36 deleted '' 2024-09-11T09:08:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_36 2024-09-11T09:08:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:36.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_36 2024-09-11T09:08:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_36 2024-09-11T09:08:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_36 2024-09-11T09:08:36.834 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.878 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_36: (2) No such file or directory 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_37 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_37 deleted 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:08:36.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_37 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_37 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-11T09:08:36.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:36.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-11T09:08:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:36.926 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:36.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_37 deleted '' 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_37 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:37.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_37 2024-09-11T09:08:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_37 2024-09-11T09:08:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_37 2024-09-11T09:08:37.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.076 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_37: (2) No such file or directory 2024-09-11T09:08:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:37.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_38 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_38 deleted 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_38 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_38 2024-09-11T09:08:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-11T09:08:37.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.119 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-11T09:08:37.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:37.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_38 deleted '' 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_38 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:37.225 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_38 2024-09-11T09:08:37.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_38 2024-09-11T09:08:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_38 2024-09-11T09:08:37.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.271 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_38: (2) No such file or directory 2024-09-11T09:08:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:37.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_39 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_39 deleted 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:37.277 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_39 2024-09-11T09:08:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_39 2024-09-11T09:08:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-11T09:08:37.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.316 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-11T09:08:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:37.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:37.421 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_39 deleted '' 2024-09-11T09:08:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_39 2024-09-11T09:08:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_39 2024-09-11T09:08:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_39 2024-09-11T09:08:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_39 2024-09-11T09:08:37.424 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_39: (2) No such file or directory 2024-09-11T09:08:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:37.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_40 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_40 deleted 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_40 2024-09-11T09:08:37.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_40 2024-09-11T09:08:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-11T09:08:37.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.511 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-11T09:08:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:37.515 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_40 deleted '' 2024-09-11T09:08:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_40 2024-09-11T09:08:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:37.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:37.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_40 2024-09-11T09:08:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_40 2024-09-11T09:08:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_40 2024-09-11T09:08:37.620 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_40: (2) No such file or directory 2024-09-11T09:08:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:37.668 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_41 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_41 deleted 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_41 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_41 2024-09-11T09:08:37.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-11T09:08:37.670 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-11T09:08:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:37.710 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_41 deleted '' 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_41 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:37.812 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_41 2024-09-11T09:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_41 2024-09-11T09:08:37.813 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_41 2024-09-11T09:08:37.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_41: (2) No such file or directory 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_42 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_42 deleted 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:08:37.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_42 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_42 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-11T09:08:37.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:37.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-11T09:08:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:37.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:38.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_42 deleted '' 2024-09-11T09:08:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_42 2024-09-11T09:08:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:38.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_42 2024-09-11T09:08:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_42 2024-09-11T09:08:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_42 2024-09-11T09:08:38.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_42: (2) No such file or directory 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_43 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_43 deleted 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_43 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_43 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-11T09:08:38.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-11T09:08:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:38.102 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_43 deleted '' 2024-09-11T09:08:38.203 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_43 2024-09-11T09:08:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:38.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_43 2024-09-11T09:08:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_43 2024-09-11T09:08:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_43 2024-09-11T09:08:38.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.256 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_43: (2) No such file or directory 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_44 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_44 deleted 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:38.262 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_44 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_44 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-11T09:08:38.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.298 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-11T09:08:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:38.302 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_44 deleted '' 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_44 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:38.404 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_44 2024-09-11T09:08:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_44 2024-09-11T09:08:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_44 2024-09-11T09:08:38.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_44: (2) No such file or directory 2024-09-11T09:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:38.458 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_45 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_45 deleted 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_45 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_45 2024-09-11T09:08:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-11T09:08:38.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.705 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-11T09:08:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:38.712 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:38.713 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_45 deleted '' 2024-09-11T09:08:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_45 2024-09-11T09:08:38.814 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:38.815 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_45 2024-09-11T09:08:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_45 2024-09-11T09:08:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_45 2024-09-11T09:08:38.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_45: (2) No such file or directory 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_46 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_46 deleted 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_46 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_46 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-11T09:08:38.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:38.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-11T09:08:38.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:38.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_46 deleted '' 2024-09-11T09:08:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_46 2024-09-11T09:08:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:39.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_46 2024-09-11T09:08:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_46 2024-09-11T09:08:39.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_46 2024-09-11T09:08:39.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_46: (2) No such file or directory 2024-09-11T09:08:39.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_47 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_47 deleted 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_47 2024-09-11T09:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_47 2024-09-11T09:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-11T09:08:39.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-11T09:08:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:39.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_47 deleted '' 2024-09-11T09:08:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_47 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_47 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_47 2024-09-11T09:08:39.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_47 2024-09-11T09:08:39.217 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_47: (2) No such file or directory 2024-09-11T09:08:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:39.473 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_48 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_48 deleted 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_48 2024-09-11T09:08:39.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_48 2024-09-11T09:08:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-11T09:08:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.519 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-11T09:08:39.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:39.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_48 deleted '' 2024-09-11T09:08:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_48 2024-09-11T09:08:39.624 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:39.625 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_48 2024-09-11T09:08:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_48 2024-09-11T09:08:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_48 2024-09-11T09:08:39.626 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_48: (2) No such file or directory 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_49 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_49 deleted 2024-09-11T09:08:39.676 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_49 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_49 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-11T09:08:39.677 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.715 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-11T09:08:39.719 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:39.720 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_49 deleted '' 2024-09-11T09:08:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_49 2024-09-11T09:08:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:39.822 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_49 2024-09-11T09:08:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_49 2024-09-11T09:08:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_49 2024-09-11T09:08:39.823 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_49: (2) No such file or directory 2024-09-11T09:08:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:39.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 ${IMAGE_COUNT}` 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_50 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_50 deleted 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_50 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_50 2024-09-11T09:08:39.872 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-11T09:08:39.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:39.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-11T09:08:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:39.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_50 deleted '' 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_50 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:40.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_50 2024-09-11T09:08:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_50 2024-09-11T09:08:40.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_50 2024-09-11T09:08:40.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:40.073 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_50: (2) No such file or directory 2024-09-11T09:08:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:40.078 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ testlog 'TEST: delete images during bootstrap' 2024-09-11T09:08:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ tee -a /tmp/tmp.LFCa6yxq8J/rbd-mirror.test.log 2024-09-11T09:08:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ date '+%F %T' 2024-09-11T09:08:40.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo 2024-09-11 09:08:40 TEST: delete images during bootstrap 2024-09-11T09:08:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:2024-09-11 09:08:40 TEST: delete images during bootstrap 2024-09-11T09:08:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_pool_mirror_mode cluster1 mirror image 2024-09-11T09:08:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=image 2024-09-11T09:08:40.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster1 mirror pool enable mirror image 2024-09-11T09:08:40.121 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd: mirroring is already configured for image mode 2024-09-11T09:08:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_pool_mirror_mode cluster2 mirror image 2024-09-11T09:08:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=image 2024-09-11T09:08:40.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror pool enable mirror image 2024-09-11T09:08:40.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd: mirroring is already configured for image mode 2024-09-11T09:08:40.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ start_mirror cluster1 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set_cluster_instance cluster1 cluster instance 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local val=cluster1 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster_var_name=cluster 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local instance_var_name=instance 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cluster=cluster1 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=cluster1 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' cluster1 = cluster1 ']' 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ instance=0 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval cluster=cluster1 2024-09-11T09:08:40.164 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ cluster=cluster1 2024-09-11T09:08:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ eval instance=0 2024-09-11T09:08:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ instance=0 2024-09-11T09:08:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:08:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:08:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=test 2024-09-11T09:08:40.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 1 10 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_1 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_1 512M 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:08:40.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:08:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:08:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:08:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:08:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:08:40.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_1 2024-09-11T09:08:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_1 2024-09-11T09:08:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:08:40.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_1 snapshot 2024-09-11T09:08:40.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:08:40.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_1 2024-09-11T09:08:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_1': 2024-09-11T09:08:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:08:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:08:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:08:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 192aa1cdb439 2024-09-11T09:08:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.192aa1cdb439 2024-09-11T09:08:40.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:08:40 2024 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:08:40 2024 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:08:40 2024 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 163a5fa4-4837-4670-84fa-fa40e770f72f 2024-09-11T09:08:40.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:08:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_1 2024-09-11T09:08:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:40.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:08:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=8 2024-09-11T09:08:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:08:40.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_1 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:08:41.100 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 42149888 bytes 2024-09-11T09:08:41.319 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.0 2024-09-11T09:08:41.319 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Sent signal 15 2024-09-11T09:08:41.320 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-11T09:08:41.320 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:08:41.323+0000 7f27c34a7640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 54737) UID: 1000 2024-09-11T09:08:41.335 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 107532288 bytes 2024-09-11T09:08:41.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 152624128 bytes 2024-09-11T09:08:41.704 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 186007552 bytes 2024-09-11T09:08:41.847 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 212147200 bytes 2024-09-11T09:08:41.940 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 219270144 bytes 2024-09-11T09:08:42.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 288159744 bytes 2024-09-11T09:08:42.355 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 314542080 bytes 2024-09-11T09:08:42.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 315166720 bytes 2024-09-11T09:08:42.656 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 374620160 bytes 2024-09-11T09:08:42.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 439949312 bytes 2024-09-11T09:08:43.106 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 514717696 bytes 2024-09-11T09:08:43.325 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 562387968 bytes 2024-09-11T09:08:43.501 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 605685760 bytes 2024-09-11T09:08:43.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 658371584 bytes 2024-09-11T09:08:43.936 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 751473664 bytes 2024-09-11T09:08:44.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 829535232 bytes 2024-09-11T09:08:44.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 844641280 bytes 2024-09-11T09:08:44.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 919339008 bytes 2024-09-11T09:08:44.709 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 937198592 bytes 2024-09-11T09:08:44.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 992066560 bytes 2024-09-11T09:08:45.110 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1039412224 bytes 2024-09-11T09:08:45.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1061353472 bytes 2024-09-11T09:08:45.407 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1123920896 bytes 2024-09-11T09:08:45.654 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1177284608 bytes 2024-09-11T09:08:45.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1196995584 bytes 2024-09-11T09:08:46.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1251264512 bytes 2024-09-11T09:08:46.205 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1298521088 bytes 2024-09-11T09:08:46.528 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1424326656 bytes 2024-09-11T09:08:46.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1483272192 bytes 2024-09-11T09:08:46.888 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1501009920 bytes 2024-09-11T09:08:47.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1523129344 bytes 2024-09-11T09:08:47.265 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1605368832 bytes 2024-09-11T09:08:47.413 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1627855872 bytes 2024-09-11T09:08:47.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1664606208 bytes 2024-09-11T09:08:47.759 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1706031104 bytes 2024-09-11T09:08:47.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1730037760 bytes 2024-09-11T09:08:48.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1791084544 bytes 2024-09-11T09:08:48.320 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.0 2024-09-11T09:08:48.321 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:08:48.321 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.0 2024-09-11T09:08:48.321 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Restarting daemon 2024-09-11T09:08:48.322 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 2024-09-11T09:08:48.324 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Started 2024-09-11T09:08:48.324 INFO:tasks.rbd_mirror_thrash:waiting for 86 secs before thrashing 2024-09-11T09:08:48.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1851560960 bytes 2024-09-11T09:08:48.419 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.4773137837997302) > weight (0.25) 2024-09-11T09:08:48.419 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.1 2024-09-11T09:08:48.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Sent signal 15 2024-09-11T09:08:48.420 INFO:tasks.rbd_mirror_thrash:waiting for 6 secs before reviving daemons 2024-09-11T09:08:48.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:08:48.423+0000 7f04bc915640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 49420) UID: 1000 2024-09-11T09:08:48.472 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1872512000 bytes 2024-09-11T09:08:48.739 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1962286080 bytes 2024-09-11T09:08:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:08:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:08:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:08:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:48.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_1 present 2024-09-11T09:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:48.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T09:08:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_1 2024-09-11T09:08:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T09:08:48.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a13237bd95 2024-09-11T09:08:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:48.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_1 present 12a13237bd95 2024-09-11T09:08:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:08:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a13237bd95 2024-09-11T09:08:49.015 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T09:08:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_1 2024-09-11T09:08:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:49.016 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T09:08:49.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a13237bd95 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a13237bd95 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a13237bd95 -o 12a13237bd95 = 12a13237bd95 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_1 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:49.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_1 2024-09-11T09:08:49.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:08:49.839+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3bbc6400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=185 2024-09-11T09:08:49.835 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:08:49.839+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3bbc6400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=187, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=b9, last_copied_object_number=128, snap_seqs={185=18446744073709551614}] 2024-09-11T09:08:54.421 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.1 2024-09-11T09:08:54.421 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:08:54.421 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.1 2024-09-11T09:08:54.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Restarting daemon 2024-09-11T09:08:54.421 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 2024-09-11T09:08:54.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Started 2024-09-11T09:08:54.425 INFO:tasks.rbd_mirror_thrash:waiting for 107 secs before thrashing 2024-09-11T09:08:54.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:08:54.846 INFO:tasks.ceph.cluster2.osd.2.smithi077.stderr:2024-09-11T09:08:54.843+0000 7f00d11de640 -1 reset not still connected to 0x55b61579a750 2024-09-11T09:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_1 deleted 2024-09-11T09:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:54.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:08:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:08:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:54.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T09:08:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_1 2024-09-11T09:08:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T09:08:54.864 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:54.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T09:08:54.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:08:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:08:54.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:08:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_1 deleted '' 2024-09-11T09:08:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:08:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:08:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:08:55.017 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:08:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:08:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_1 2024-09-11T09:08:55.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:08:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:08:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_1 2024-09-11T09:08:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_1 2024-09-11T09:08:55.019 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:08:55.062 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_1: (2) No such file or directory 2024-09-11T09:08:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:08:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:08:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:08:55.066 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:08:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_1 2024-09-11T09:08:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:55.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_1 2024-09-11T09:08:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_1 2024-09-11T09:08:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:08:55.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:08:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_1 2024-09-11T09:08:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:08:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:08:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_1 2024-09-11T09:08:55.117 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_1 2024-09-11T09:08:55.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_1 2024-09-11T09:09:18.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_2 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_2 512M 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:09:18.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:09:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:09:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:09:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:09:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:09:18.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_2 2024-09-11T09:09:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_2 2024-09-11T09:09:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:09:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:09:18.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_2 snapshot 2024-09-11T09:09:18.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:09:18.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_2 2024-09-11T09:09:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_2': 2024-09-11T09:09:18.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 19505d44b747 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.19505d44b747 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:09:18 2024 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:09:18 2024 2024-09-11T09:09:18.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:09:18 2024 2024-09-11T09:09:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:09:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:09:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 163f9c1e-b599-4b0f-a951-2d945cb90cf4 2024-09-11T09:09:18.969 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:09:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_2 2024-09-11T09:09:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:09:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:18.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:09:18.973 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=12 2024-09-11T09:09:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:09:18.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 12s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_2 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:09:19.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 40790016 bytes 2024-09-11T09:09:19.307 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 59922432 bytes 2024-09-11T09:09:19.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 107788288 bytes 2024-09-11T09:09:19.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 147383296 bytes 2024-09-11T09:09:19.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 163162112 bytes 2024-09-11T09:09:19.877 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 178874368 bytes 2024-09-11T09:09:20.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 249334784 bytes 2024-09-11T09:09:20.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 277441536 bytes 2024-09-11T09:09:20.418 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 320487424 bytes 2024-09-11T09:09:20.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 377771008 bytes 2024-09-11T09:09:20.974 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 416207872 bytes 2024-09-11T09:09:21.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 520519680 bytes 2024-09-11T09:09:21.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 562148352 bytes 2024-09-11T09:09:21.644 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 603735040 bytes 2024-09-11T09:09:21.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 633461760 bytes 2024-09-11T09:09:21.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 668275712 bytes 2024-09-11T09:09:22.193 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 742110208 bytes 2024-09-11T09:09:22.337 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 765263872 bytes 2024-09-11T09:09:22.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 843620352 bytes 2024-09-11T09:09:22.761 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 896599040 bytes 2024-09-11T09:09:22.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 955529216 bytes 2024-09-11T09:09:23.177 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1012884480 bytes 2024-09-11T09:09:23.308 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1040818176 bytes 2024-09-11T09:09:23.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1079855104 bytes 2024-09-11T09:09:23.593 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1084360704 bytes 2024-09-11T09:09:23.789 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1130554368 bytes 2024-09-11T09:09:23.972 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1179200512 bytes 2024-09-11T09:09:24.216 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1259387904 bytes 2024-09-11T09:09:24.312 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1260004352 bytes 2024-09-11T09:09:24.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1322780672 bytes 2024-09-11T09:09:24.768 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1401435136 bytes 2024-09-11T09:09:24.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1437598720 bytes 2024-09-11T09:09:25.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1462282240 bytes 2024-09-11T09:09:25.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1462875136 bytes 2024-09-11T09:09:25.338 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1493095424 bytes 2024-09-11T09:09:25.460 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1504638976 bytes 2024-09-11T09:09:25.633 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1551971328 bytes 2024-09-11T09:09:25.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1653335040 bytes 2024-09-11T09:09:26.194 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1738309632 bytes 2024-09-11T09:09:26.377 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1770234880 bytes 2024-09-11T09:09:26.545 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1804721152 bytes 2024-09-11T09:09:26.700 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1830382592 bytes 2024-09-11T09:09:26.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1857883136 bytes 2024-09-11T09:09:26.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1882305536 bytes 2024-09-11T09:09:27.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1904096256 bytes 2024-09-11T09:09:27.401 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1976080384 bytes 2024-09-11T09:09:27.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2034605056 bytes 2024-09-11T09:09:27.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2132071424 bytes 2024-09-11T09:09:28.067 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2162825216 bytes 2024-09-11T09:09:28.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2215556096 bytes 2024-09-11T09:09:28.407 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2246790144 bytes 2024-09-11T09:09:28.621 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2292955136 bytes 2024-09-11T09:09:28.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2321512448 bytes 2024-09-11T09:09:28.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2322112512 bytes 2024-09-11T09:09:28.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2344733696 bytes 2024-09-11T09:09:29.162 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2377939968 bytes 2024-09-11T09:09:29.379 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2444354560 bytes 2024-09-11T09:09:29.561 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2502494208 bytes 2024-09-11T09:09:29.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2519395328 bytes 2024-09-11T09:09:29.860 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2538066944 bytes 2024-09-11T09:09:30.113 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2626131968 bytes 2024-09-11T09:09:30.299 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2659875840 bytes 2024-09-11T09:09:30.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2660513792 bytes 2024-09-11T09:09:30.541 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2685681664 bytes 2024-09-11T09:09:30.689 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2732713984 bytes 2024-09-11T09:09:30.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2777572352 bytes 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_2 present 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:30.979 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_2 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T09:09:30.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:09:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a1c725d2f2 2024-09-11T09:09:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:09:31.032 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:09:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_2 present 12a1c725d2f2 2024-09-11T09:09:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:09:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:31.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a1c725d2f2 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_2 2024-09-11T09:09:31.134 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T09:09:31.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:09:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a1c725d2f2 2024-09-11T09:09:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a1c725d2f2 2024-09-11T09:09:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a1c725d2f2 -o 12a1c725d2f2 = 12a1c725d2f2 2024-09-11T09:09:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:09:31.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:09:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:09:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_2 2024-09-11T09:09:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:09:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:31.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_2 2024-09-11T09:09:31.972 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:09:31.974+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3b190400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=186 2024-09-11T09:09:31.973 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:09:31.974+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3b190400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=188, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=ba, last_copied_object_number=128, snap_seqs={186=18446744073709551614}] 2024-09-11T09:09:34.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:09:34.710+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4fcbd40 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:09:34.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:09:34.710+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3af00 [3/163f9c1e-b599-4b0f-a951-2d945cb90cf4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:09:35.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:09:35.518+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4fcb040 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:09:35.517 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:09:35.518+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3af00 [3/163f9c1e-b599-4b0f-a951-2d945cb90cf4] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:09:36.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_2 deleted 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:09:36.989 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:09:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T09:09:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:09:36.990 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:09:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_2 2024-09-11T09:09:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T09:09:36.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:09:37.033 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T09:09:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:09:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:09:37.037 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_2 deleted '' 2024-09-11T09:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:09:37.139 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:09:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:09:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_2 2024-09-11T09:09:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:09:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:09:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_2 2024-09-11T09:09:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_2 2024-09-11T09:09:37.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:09:37.185 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_2: (2) No such file or directory 2024-09-11T09:09:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:09:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:09:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:09:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:09:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_2 2024-09-11T09:09:37.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:09:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:37.191 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_2 2024-09-11T09:09:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_2 2024-09-11T09:09:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:09:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:09:37.241 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:09:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_2 2024-09-11T09:09:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:09:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:09:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_2 2024-09-11T09:09:37.242 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_2 2024-09-11T09:09:37.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_2 2024-09-11T09:10:01.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:10:01.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_3 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_3 512M 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:10:01.093 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:10:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:01.094 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_3 2024-09-11T09:10:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_3 2024-09-11T09:10:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:01.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:10:01.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_3 snapshot 2024-09-11T09:10:01.793 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:10:01.810 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_3 2024-09-11T09:10:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_3': 2024-09-11T09:10:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:10:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:10:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:10:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1968dc1694f0 2024-09-11T09:10:02.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1968dc1694f0 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:10:01 2024 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:10:01 2024 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:10:01 2024 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: ad047461-d7be-4a5d-8330-522995f0596f 2024-09-11T09:10:02.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:10:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_3 2024-09-11T09:10:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:02.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:02.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:10:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=9 2024-09-11T09:10:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:10:02.072 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 9s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_3 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:10:02.279 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 18071552 bytes 2024-09-11T09:10:02.470 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 74432512 bytes 2024-09-11T09:10:02.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 124700672 bytes 2024-09-11T09:10:02.859 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 167234560 bytes 2024-09-11T09:10:02.986 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 187580416 bytes 2024-09-11T09:10:03.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 256033792 bytes 2024-09-11T09:10:03.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 329721856 bytes 2024-09-11T09:10:03.544 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 338949120 bytes 2024-09-11T09:10:03.725 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 384813056 bytes 2024-09-11T09:10:03.970 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 464022528 bytes 2024-09-11T09:10:04.309 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 594989056 bytes 2024-09-11T09:10:04.480 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 636614656 bytes 2024-09-11T09:10:04.703 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 712284160 bytes 2024-09-11T09:10:04.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 729435136 bytes 2024-09-11T09:10:04.968 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 752386048 bytes 2024-09-11T09:10:05.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 765463552 bytes 2024-09-11T09:10:05.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 828795904 bytes 2024-09-11T09:10:05.549 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 920838144 bytes 2024-09-11T09:10:05.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 964192256 bytes 2024-09-11T09:10:05.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 983860224 bytes 2024-09-11T09:10:05.985 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1001980928 bytes 2024-09-11T09:10:06.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1036354560 bytes 2024-09-11T09:10:06.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1091195904 bytes 2024-09-11T09:10:06.585 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1169353728 bytes 2024-09-11T09:10:06.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1177869312 bytes 2024-09-11T09:10:06.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1206489088 bytes 2024-09-11T09:10:06.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1237252096 bytes 2024-09-11T09:10:07.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1259237376 bytes 2024-09-11T09:10:07.314 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1308533760 bytes 2024-09-11T09:10:07.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1340878848 bytes 2024-09-11T09:10:07.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1372844032 bytes 2024-09-11T09:10:07.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1416545280 bytes 2024-09-11T09:10:08.024 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1472758784 bytes 2024-09-11T09:10:08.318 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1595183104 bytes 2024-09-11T09:10:08.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1606036480 bytes 2024-09-11T09:10:08.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1677146112 bytes 2024-09-11T09:10:08.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1692109824 bytes 2024-09-11T09:10:09.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1757363200 bytes 2024-09-11T09:10:09.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1791217664 bytes 2024-09-11T09:10:09.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1820402688 bytes 2024-09-11T09:10:09.467 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1829482496 bytes 2024-09-11T09:10:09.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1895704576 bytes 2024-09-11T09:10:09.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1912098816 bytes 2024-09-11T09:10:10.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1939010560 bytes 2024-09-11T09:10:10.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 2043723776 bytes 2024-09-11T09:10:10.530 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 2110048256 bytes 2024-09-11T09:10:10.699 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 2144805888 bytes 2024-09-11T09:10:10.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 2163372032 bytes 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_3 present 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:11.079 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_3 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T09:10:11.080 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a1be2c297a 2024-09-11T09:10:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:10:11.133 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:10:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_3 present 12a1be2c297a 2024-09-11T09:10:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:11.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a1be2c297a 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_3 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T09:10:11.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:11.296 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a1be2c297a 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a1be2c297a 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a1be2c297a -o 12a1be2c297a = 12a1be2c297a 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_3 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:11.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_3 2024-09-11T09:10:11.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:10:11.885+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3c1e0c00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=187 2024-09-11T09:10:11.886 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:10:11.885+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3c1e0c00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=189, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=bb, last_copied_object_number=128, snap_seqs={187=18446744073709551614}] 2024-09-11T09:10:14.325 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.4718949607002869) > weight (0.25) 2024-09-11T09:10:14.325 INFO:tasks.rbd_mirror_thrash:kill cluster2.client.mirror.1 2024-09-11T09:10:14.325 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Sent signal 15 2024-09-11T09:10:14.326 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-11T09:10:14.326 INFO:tasks.rbd_mirror_thrash:waiting for cluster2.client.mirror.1 2024-09-11T09:10:14.326 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:10:14.327 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:10:14.325+0000 7f3bd7179640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 44429) UID: 1000 2024-09-11T09:10:15.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:10:15.417+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4fca680 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:10:15.418 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:10:15.417+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3b680 [3/ad047461-d7be-4a5d-8330-522995f0596f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:10:16.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:10:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_3 deleted 2024-09-11T09:10:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:16.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_3 2024-09-11T09:10:16.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T09:10:16.901 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:16.951 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T09:10:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:10:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:10:16.956 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:10:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_3 deleted '' 2024-09-11T09:10:17.057 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:10:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:10:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:10:17.058 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_3 2024-09-11T09:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_3 2024-09-11T09:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_3 2024-09-11T09:10:17.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:17.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_3: (2) No such file or directory 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_3 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:17.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_3 2024-09-11T09:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_3 2024-09-11T09:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:10:17.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_3 2024-09-11T09:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_3 2024-09-11T09:10:17.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_3 2024-09-11T09:10:17.210 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_3 2024-09-11T09:10:20.428 INFO:tasks.rbd_mirror_thrash:reviving cluster2.client.mirror.1 2024-09-11T09:10:20.428 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Restarting daemon 2024-09-11T09:10:20.428 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 2024-09-11T09:10:20.432 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Started 2024-09-11T09:10:20.432 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-11T09:10:38.126 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_4 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_4 512M 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:10:38.131 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:10:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:10:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:10:38.132 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_4 2024-09-11T09:10:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_4 2024-09-11T09:10:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:10:38.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_4 snapshot 2024-09-11T09:10:38.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:10:38.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_4 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_4': 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1989261d1864 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1989261d1864 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:10:38.862 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:10:38 2024 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:10:38 2024 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:10:38 2024 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: eb5a3aa4-034d-41eb-bec8-1cf1f0bf9338 2024-09-11T09:10:38.863 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:10:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_4 2024-09-11T09:10:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:38.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:10:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=6 2024-09-11T09:10:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:10:38.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 6s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_4 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:10:39.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 65725440 bytes 2024-09-11T09:10:39.313 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 113736704 bytes 2024-09-11T09:10:39.475 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 162958336 bytes 2024-09-11T09:10:39.648 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 189545472 bytes 2024-09-11T09:10:39.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 250408960 bytes 2024-09-11T09:10:40.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 307081216 bytes 2024-09-11T09:10:40.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 372336640 bytes 2024-09-11T09:10:40.447 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 406609920 bytes 2024-09-11T09:10:40.638 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 441063424 bytes 2024-09-11T09:10:40.821 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 490233856 bytes 2024-09-11T09:10:40.924 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 495227904 bytes 2024-09-11T09:10:41.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 517082112 bytes 2024-09-11T09:10:41.257 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 581047296 bytes 2024-09-11T09:10:41.426 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T09:10:41.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T09:10:41.426 INFO:tasks.rbd_mirror_thrash:waiting for 4 secs before reviving daemons 2024-09-11T09:10:41.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:10:41.428+0000 7fe56e00b640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 52726) UID: 1000 2024-09-11T09:10:41.438 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 632731648 bytes 2024-09-11T09:10:41.583 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 639732736 bytes 2024-09-11T09:10:41.781 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 696495104 bytes 2024-09-11T09:10:41.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 747146240 bytes 2024-09-11T09:10:42.154 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 816548864 bytes 2024-09-11T09:10:42.293 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 840682496 bytes 2024-09-11T09:10:42.485 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 888903680 bytes 2024-09-11T09:10:42.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 967441408 bytes 2024-09-11T09:10:42.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1039673344 bytes 2024-09-11T09:10:43.096 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1059457024 bytes 2024-09-11T09:10:43.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1063463936 bytes 2024-09-11T09:10:43.490 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1145218048 bytes 2024-09-11T09:10:43.724 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1210585088 bytes 2024-09-11T09:10:43.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1253364736 bytes 2024-09-11T09:10:44.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1284739072 bytes 2024-09-11T09:10:44.317 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1369302016 bytes 2024-09-11T09:10:44.556 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1440444416 bytes 2024-09-11T09:10:44.663 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1461220352 bytes 2024-09-11T09:10:44.873 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_4 present 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:10:44.874 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T09:10:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_4 2024-09-11T09:10:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T09:10:44.875 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a14014a78e 2024-09-11T09:10:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:10:44.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:10:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_4 present 12a14014a78e 2024-09-11T09:10:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:10:45.028 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a14014a78e 2024-09-11T09:10:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:10:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:10:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T09:10:45.029 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_4 2024-09-11T09:10:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T09:10:45.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a14014a78e 2024-09-11T09:10:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a14014a78e 2024-09-11T09:10:45.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a14014a78e -o 12a14014a78e = 12a14014a78e 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_4 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:45.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_4 2024-09-11T09:10:45.427 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T09:10:45.427 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:10:45.428 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T09:10:45.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T09:10:45.428 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T09:10:45.431 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T09:10:45.431 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-11T09:10:45.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:10:45.864+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3af2dc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=188 2024-09-11T09:10:45.867 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:10:45.864+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3af2dc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=190, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=bc, last_copied_object_number=128, snap_seqs={188=18446744073709551614}] 2024-09-11T09:10:50.868 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_4 deleted 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:10:50.883 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T09:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_4 2024-09-11T09:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T09:10:50.884 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:50.933 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T09:10:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:10:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:10:50.937 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:10:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_4 deleted '' 2024-09-11T09:10:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:10:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:51.039 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:10:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:10:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:10:51.040 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:10:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_4 2024-09-11T09:10:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:10:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:10:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_4 2024-09-11T09:10:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_4 2024-09-11T09:10:51.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:10:51.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_4: (2) No such file or directory 2024-09-11T09:10:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:10:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:10:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:10:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:10:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_4 2024-09-11T09:10:51.091 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:51.092 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_4 2024-09-11T09:10:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_4 2024-09-11T09:10:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:10:51.140 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_4 2024-09-11T09:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_4 2024-09-11T09:10:51.141 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_4 2024-09-11T09:10:51.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_4 2024-09-11T09:11:15.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:11:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:11:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:11:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_5 2024-09-11T09:11:15.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_5 512M 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:15.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_5 2024-09-11T09:11:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_5 2024-09-11T09:11:15.286 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:11:15.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_5 snapshot 2024-09-11T09:11:15.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:11:15.949 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_5 2024-09-11T09:11:15.995 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_5': 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 19a1db12b774 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.19a1db12b774 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:11:15 2024 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:11:15 2024 2024-09-11T09:11:15.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:11:15 2024 2024-09-11T09:11:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:11:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:11:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 5b801bda-e2c4-4b2d-9ee2-802a0b7cf322 2024-09-11T09:11:15.997 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:11:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_5 2024-09-11T09:11:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:16.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:11:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=13 2024-09-11T09:11:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:11:16.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 13s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_5 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:11:16.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 33690624 bytes 2024-09-11T09:11:16.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 94522368 bytes 2024-09-11T09:11:16.569 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 125412352 bytes 2024-09-11T09:11:16.785 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 186424320 bytes 2024-09-11T09:11:16.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 222854144 bytes 2024-09-11T09:11:17.116 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 259466240 bytes 2024-09-11T09:11:17.232 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 276807680 bytes 2024-09-11T09:11:17.406 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 326594560 bytes 2024-09-11T09:11:17.516 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 340461568 bytes 2024-09-11T09:11:17.679 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 374695936 bytes 2024-09-11T09:11:17.839 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 419544064 bytes 2024-09-11T09:11:17.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 431589376 bytes 2024-09-11T09:11:18.163 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 489380864 bytes 2024-09-11T09:11:18.330 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 526405632 bytes 2024-09-11T09:11:18.446 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 541358080 bytes 2024-09-11T09:11:18.586 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 567096320 bytes 2024-09-11T09:11:18.767 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 630920192 bytes 2024-09-11T09:11:18.984 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 703002624 bytes 2024-09-11T09:11:19.145 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 734316544 bytes 2024-09-11T09:11:19.431 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 818390016 bytes 2024-09-11T09:11:19.547 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 853018624 bytes 2024-09-11T09:11:19.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 859948032 bytes 2024-09-11T09:11:19.837 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 885199872 bytes 2024-09-11T09:11:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 887661568 bytes 2024-09-11T09:11:20.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 896973824 bytes 2024-09-11T09:11:20.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 950550528 bytes 2024-09-11T09:11:20.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 963257344 bytes 2024-09-11T09:11:20.563 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1013599232 bytes 2024-09-11T09:11:20.756 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1056297984 bytes 2024-09-11T09:11:20.945 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1110541312 bytes 2024-09-11T09:11:21.171 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1195588608 bytes 2024-09-11T09:11:21.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1262411776 bytes 2024-09-11T09:11:21.646 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1340613632 bytes 2024-09-11T09:11:21.801 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1372882944 bytes 2024-09-11T09:11:22.059 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1454622720 bytes 2024-09-11T09:11:22.236 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1489103872 bytes 2024-09-11T09:11:22.426 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1542995968 bytes 2024-09-11T09:11:22.578 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1577586688 bytes 2024-09-11T09:11:22.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1601051648 bytes 2024-09-11T09:11:22.899 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1636189184 bytes 2024-09-11T09:11:23.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1674531840 bytes 2024-09-11T09:11:23.218 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1720738816 bytes 2024-09-11T09:11:23.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1745414144 bytes 2024-09-11T09:11:23.551 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1776820224 bytes 2024-09-11T09:11:23.744 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1824645120 bytes 2024-09-11T09:11:23.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1839972352 bytes 2024-09-11T09:11:24.034 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1877760000 bytes 2024-09-11T09:11:24.204 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1918304256 bytes 2024-09-11T09:11:24.341 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 1945293824 bytes 2024-09-11T09:11:24.495 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 1977091072 bytes 2024-09-11T09:11:24.651 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2007972864 bytes 2024-09-11T09:11:24.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2044301312 bytes 2024-09-11T09:11:24.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2049942528 bytes 2024-09-11T09:11:25.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2069450752 bytes 2024-09-11T09:11:25.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2070054912 bytes 2024-09-11T09:11:25.261 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5600 IOs, 2076620800 bytes 2024-09-11T09:11:25.434 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5700 IOs, 2123346944 bytes 2024-09-11T09:11:25.588 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5800 IOs, 2154853376 bytes 2024-09-11T09:11:25.760 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5900 IOs, 2179548160 bytes 2024-09-11T09:11:25.900 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6000 IOs, 2205923328 bytes 2024-09-11T09:11:26.074 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6100 IOs, 2241772544 bytes 2024-09-11T09:11:26.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6200 IOs, 2255980544 bytes 2024-09-11T09:11:26.417 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6300 IOs, 2320143360 bytes 2024-09-11T09:11:26.517 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6400 IOs, 2320749568 bytes 2024-09-11T09:11:26.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6500 IOs, 2383859712 bytes 2024-09-11T09:11:26.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6600 IOs, 2426631168 bytes 2024-09-11T09:11:27.046 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6700 IOs, 2432688128 bytes 2024-09-11T09:11:27.259 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6800 IOs, 2492873728 bytes 2024-09-11T09:11:27.502 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:6900 IOs, 2558465024 bytes 2024-09-11T09:11:27.718 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7000 IOs, 2610510848 bytes 2024-09-11T09:11:27.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7100 IOs, 2640955392 bytes 2024-09-11T09:11:28.071 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7200 IOs, 2699156480 bytes 2024-09-11T09:11:28.263 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7300 IOs, 2736893952 bytes 2024-09-11T09:11:28.471 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7400 IOs, 2797530112 bytes 2024-09-11T09:11:28.629 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7500 IOs, 2826038272 bytes 2024-09-11T09:11:28.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7600 IOs, 2892806144 bytes 2024-09-11T09:11:28.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:7700 IOs, 2893414400 bytes 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_5 present 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:11:29.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:11:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T09:11:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:11:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:11:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_5 2024-09-11T09:11:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T09:11:29.010 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:11:29.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a191caf179 2024-09-11T09:11:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:11:29.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:11:29.165 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_5 present 12a191caf179 2024-09-11T09:11:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:11:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:11:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a191caf179 2024-09-11T09:11:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:11:29.166 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:11:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T09:11:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:11:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:11:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_5 2024-09-11T09:11:29.167 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T09:11:29.168 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:11:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a191caf179 2024-09-11T09:11:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a191caf179 2024-09-11T09:11:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a191caf179 -o 12a191caf179 = 12a191caf179 2024-09-11T09:11:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:11:29.223 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:11:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:11:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_5 2024-09-11T09:11:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:29.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_5 2024-09-11T09:11:29.432 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7149272725188947) > weight (0.25) 2024-09-11T09:11:29.433 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.9611903651327659) > weight (0.25) 2024-09-11T09:11:29.433 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.29703568581456663) > weight (0.25) 2024-09-11T09:11:29.433 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.4980001357598931) > weight (0.25) 2024-09-11T09:11:29.433 INFO:tasks.rbd_mirror_thrash:waiting for 69 secs before thrashing 2024-09-11T09:11:30.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:11:30.011+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3ea2fc00 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=189 2024-09-11T09:11:30.011 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:11:30.011+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3ea2fc00 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=191, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=bd, last_copied_object_number=128, snap_seqs={189=18446744073709551614}] 2024-09-11T09:11:34.709 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:11:34.711+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4fcb860 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:11:34.710 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:11:34.711+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4670000 [3/5b801bda-e2c4-4b2d-9ee2-802a0b7cf322] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:11:35.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:11:35.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_5 deleted 2024-09-11T09:11:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:11:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:11:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:11:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:11:35.013 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:11:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T09:11:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:11:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:11:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_5 2024-09-11T09:11:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T09:11:35.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:11:35.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T09:11:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:11:35.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:11:35.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:11:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_5 deleted '' 2024-09-11T09:11:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:11:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:11:35.187 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:11:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:11:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:11:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_5 2024-09-11T09:11:35.188 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:11:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:11:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_5 2024-09-11T09:11:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_5 2024-09-11T09:11:35.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:11:35.233 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_5: (2) No such file or directory 2024-09-11T09:11:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:11:35.237 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_5 2024-09-11T09:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:35.238 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_5 2024-09-11T09:11:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_5 2024-09-11T09:11:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:11:35.287 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:11:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_5 2024-09-11T09:11:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_5 2024-09-11T09:11:35.288 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_5 2024-09-11T09:11:35.334 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_5 2024-09-11T09:11:54.432 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T09:11:54.433 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T09:11:54.433 INFO:tasks.rbd_mirror_thrash:waiting for 0 secs before reviving daemons 2024-09-11T09:11:54.433 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T09:11:54.433 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:11:54.434 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:11:54.435+0000 7f37c9e73640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 64369) UID: 1000 2024-09-11T09:11:58.368 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_6 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_6 512M 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:11:58.373 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:11:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:11:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:11:58.374 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_6 2024-09-11T09:11:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_6 2024-09-11T09:11:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:11:58.422 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:11:58.423 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_6 snapshot 2024-09-11T09:11:59.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:11:59.075 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_6 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_6': 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 19b21a5b1963 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.19b21a5b1963 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:11:59.123 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:11:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:11:58 2024 2024-09-11T09:11:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:11:58 2024 2024-09-11T09:11:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:11:58 2024 2024-09-11T09:11:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:11:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:11:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 1fc421ee-ff98-4e3c-9e7f-a9261867e42f 2024-09-11T09:11:59.124 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:11:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_6 2024-09-11T09:11:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:11:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:11:59.127 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:11:59.128 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:11:59.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=5 2024-09-11T09:11:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:11:59.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 5s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_6 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:11:59.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 121461760 bytes 2024-09-11T09:11:59.671 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 161011712 bytes 2024-09-11T09:11:59.827 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 188289024 bytes 2024-09-11T09:12:00.051 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 248020992 bytes 2024-09-11T09:12:00.211 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 278759424 bytes 2024-09-11T09:12:00.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 317548544 bytes 2024-09-11T09:12:00.535 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T09:12:00.535 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T09:12:00.535 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T09:12:00.538 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T09:12:00.538 INFO:tasks.rbd_mirror_thrash:waiting for 84 secs before thrashing 2024-09-11T09:12:00.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 366904320 bytes 2024-09-11T09:12:00.702 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 381204480 bytes 2024-09-11T09:12:00.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 483478528 bytes 2024-09-11T09:12:01.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 573529088 bytes 2024-09-11T09:12:01.392 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 594739200 bytes 2024-09-11T09:12:01.553 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 630773760 bytes 2024-09-11T09:12:01.833 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 727903232 bytes 2024-09-11T09:12:02.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 780790784 bytes 2024-09-11T09:12:02.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 783438848 bytes 2024-09-11T09:12:02.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 819402752 bytes 2024-09-11T09:12:02.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 859061248 bytes 2024-09-11T09:12:02.591 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 873869312 bytes 2024-09-11T09:12:02.866 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 964465664 bytes 2024-09-11T09:12:03.012 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 992747520 bytes 2024-09-11T09:12:03.215 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1051789312 bytes 2024-09-11T09:12:03.444 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1161836544 bytes 2024-09-11T09:12:03.655 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1187604480 bytes 2024-09-11T09:12:03.786 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1227573248 bytes 2024-09-11T09:12:04.008 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1268886528 bytes 2024-09-11T09:12:04.135 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_6 present 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T09:12:04.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_6 2024-09-11T09:12:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T09:12:04.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a1fb252a8c 2024-09-11T09:12:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:12:04.189 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:12:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_6 present 12a1fb252a8c 2024-09-11T09:12:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:04.290 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:12:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a1fb252a8c 2024-09-11T09:12:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:12:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:12:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T09:12:04.291 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_6 2024-09-11T09:12:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T09:12:04.292 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a1fb252a8c 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a1fb252a8c 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a1fb252a8c -o 12a1fb252a8c = 12a1fb252a8c 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_6 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:04.349 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:04.350 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_6 2024-09-11T09:12:04.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:12:04.835+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3ea06400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=190 2024-09-11T09:12:04.832 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:12:04.835+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3ea06400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=192, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=be, last_copied_object_number=128, snap_seqs={190=18446744073709551614}] 2024-09-11T09:12:05.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:12:05.519+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece5146340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:12:05.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:12:05.519+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4670000 [3/1fc421ee-ff98-4e3c-9e7f-a9261867e42f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:12:09.830 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_6 deleted 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:12:09.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:12:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T09:12:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_6 2024-09-11T09:12:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T09:12:09.844 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:09.893 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T09:12:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:12:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:12:09.897 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_6 deleted '' 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:12:09.999 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:12:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_6 2024-09-11T09:12:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_6 2024-09-11T09:12:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_6 2024-09-11T09:12:10.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:10.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_6: (2) No such file or directory 2024-09-11T09:12:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:12:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:12:10.053 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:12:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:12:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_6 2024-09-11T09:12:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:10.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_6 2024-09-11T09:12:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_6 2024-09-11T09:12:10.103 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:12:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:12:10.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_6 2024-09-11T09:12:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_6 2024-09-11T09:12:10.105 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_6 2024-09-11T09:12:10.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_6 2024-09-11T09:12:34.144 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_7 2024-09-11T09:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_7 512M 2024-09-11T09:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:12:34.148 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:12:34.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_7 2024-09-11T09:12:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_7 2024-09-11T09:12:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:12:34.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_7 snapshot 2024-09-11T09:12:34.841 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:12:34.856 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_7 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_7': 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 19cc32befb67 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.19cc32befb67 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:12:34 2024 2024-09-11T09:12:34.902 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:12:34 2024 2024-09-11T09:12:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:12:34 2024 2024-09-11T09:12:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:12:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:12:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 2b54c4e5-c27b-4396-859b-cdd7f9a1f4cf 2024-09-11T09:12:34.903 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:12:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_7 2024-09-11T09:12:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:34.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:12:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T09:12:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:12:34.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_7 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:12:35.136 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 42291200 bytes 2024-09-11T09:12:35.282 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 67869696 bytes 2024-09-11T09:12:35.445 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 105244672 bytes 2024-09-11T09:12:35.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 142542848 bytes 2024-09-11T09:12:35.896 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 244887552 bytes 2024-09-11T09:12:35.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 245480448 bytes 2024-09-11T09:12:36.149 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 282271744 bytes 2024-09-11T09:12:36.278 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 306947072 bytes 2024-09-11T09:12:36.463 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 363042816 bytes 2024-09-11T09:12:36.602 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 390372352 bytes 2024-09-11T09:12:36.706 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 398859264 bytes 2024-09-11T09:12:37.022 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 516323328 bytes 2024-09-11T09:12:37.240 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 585954304 bytes 2024-09-11T09:12:37.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 621368320 bytes 2024-09-11T09:12:37.572 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 643354624 bytes 2024-09-11T09:12:37.836 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 731461632 bytes 2024-09-11T09:12:38.036 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 793269248 bytes 2024-09-11T09:12:38.198 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 823816192 bytes 2024-09-11T09:12:38.362 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 859853824 bytes 2024-09-11T09:12:38.433 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.7798833418055998) > weight (0.25) 2024-09-11T09:12:38.434 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.6287235454136323) > weight (0.25) 2024-09-11T09:12:38.434 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.9990442391988505) > weight (0.25) 2024-09-11T09:12:38.434 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.7190618793663539) > weight (0.25) 2024-09-11T09:12:38.434 INFO:tasks.rbd_mirror_thrash:waiting for 99 secs before thrashing 2024-09-11T09:12:38.474 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 870294528 bytes 2024-09-11T09:12:38.635 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 904876032 bytes 2024-09-11T09:12:38.734 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 908205056 bytes 2024-09-11T09:12:38.867 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 950007808 bytes 2024-09-11T09:12:39.030 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 968264704 bytes 2024-09-11T09:12:39.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 968882176 bytes 2024-09-11T09:12:39.363 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1040527360 bytes 2024-09-11T09:12:39.525 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1090422784 bytes 2024-09-11T09:12:39.696 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1115238400 bytes 2024-09-11T09:12:39.854 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1147377664 bytes 2024-09-11T09:12:40.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1202748416 bytes 2024-09-11T09:12:40.170 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1220493312 bytes 2024-09-11T09:12:40.320 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1245877248 bytes 2024-09-11T09:12:40.513 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1297782784 bytes 2024-09-11T09:12:40.680 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1342272512 bytes 2024-09-11T09:12:40.855 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1375631360 bytes 2024-09-11T09:12:41.052 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1430871040 bytes 2024-09-11T09:12:41.200 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1451830272 bytes 2024-09-11T09:12:41.491 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1544233984 bytes 2024-09-11T09:12:41.669 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1587651584 bytes 2024-09-11T09:12:41.891 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1650627584 bytes 2024-09-11T09:12:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:12:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:12:41.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_7 present 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:41.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_7 2024-09-11T09:12:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T09:12:41.917 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:41.966 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a19f853c3e 2024-09-11T09:12:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:12:41.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:12:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_7 present 12a19f853c3e 2024-09-11T09:12:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:42.068 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:12:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a19f853c3e 2024-09-11T09:12:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:12:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:12:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T09:12:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:42.069 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_7 2024-09-11T09:12:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T09:12:42.070 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a19f853c3e 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a19f853c3e 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a19f853c3e -o 12a19f853c3e = 12a19f853c3e 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_7 2024-09-11T09:12:42.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:42.130 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_7 2024-09-11T09:12:42.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:12:42.894+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3bf90400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=191 2024-09-11T09:12:42.892 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:12:42.894+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3bf90400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=193, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=bf, last_copied_object_number=128, snap_seqs={191=18446744073709551614}] 2024-09-11T09:12:47.892 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:12:47.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_7 deleted 2024-09-11T09:12:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:12:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:12:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:12:47.909 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:12:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T09:12:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_7 2024-09-11T09:12:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T09:12:47.910 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:47.954 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T09:12:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:12:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:12:47.959 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:12:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_7 deleted '' 2024-09-11T09:12:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:12:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:48.060 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:12:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:12:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:12:48.061 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:12:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_7 2024-09-11T09:12:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:12:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:12:48.064 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_7 2024-09-11T09:12:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_7 2024-09-11T09:12:48.065 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:12:48.104 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_7: (2) No such file or directory 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_7 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:48.109 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_7 2024-09-11T09:12:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_7 2024-09-11T09:12:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:48.159 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:12:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:12:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_7 2024-09-11T09:12:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:12:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:12:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_7 2024-09-11T09:12:48.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_7 2024-09-11T09:12:48.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_7 2024-09-11T09:13:12.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:13:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:13:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:13:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_8 2024-09-11T09:13:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_8 512M 2024-09-11T09:13:12.156 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:12.157 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_8 2024-09-11T09:13:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_8 2024-09-11T09:13:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:12.206 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:13:12.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_8 snapshot 2024-09-11T09:13:12.853 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:13:12.871 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_8 2024-09-11T09:13:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_8': 2024-09-11T09:13:12.920 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 19de24b48ced 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.19de24b48ced 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:13:12 2024 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:13:12 2024 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:13:12 2024 2024-09-11T09:13:12.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:13:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:13:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: cedede9a-2252-4c4f-a717-c6f92a4fee0a 2024-09-11T09:13:12.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:13:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_8 2024-09-11T09:13:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:12.925 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:13:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=7 2024-09-11T09:13:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:13:12.927 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 7s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_8 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:13:13.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 10801152 bytes 2024-09-11T09:13:13.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 30706688 bytes 2024-09-11T09:13:13.399 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 68890624 bytes 2024-09-11T09:13:13.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 85306368 bytes 2024-09-11T09:13:13.753 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 150624256 bytes 2024-09-11T09:13:13.929 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 196715520 bytes 2024-09-11T09:13:14.047 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 220859392 bytes 2024-09-11T09:13:14.235 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 254959616 bytes 2024-09-11T09:13:14.396 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 302806016 bytes 2024-09-11T09:13:14.565 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 334580736 bytes 2024-09-11T09:13:14.832 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 421467136 bytes 2024-09-11T09:13:14.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 423547904 bytes 2024-09-11T09:13:15.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 457937920 bytes 2024-09-11T09:13:15.227 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 482383872 bytes 2024-09-11T09:13:15.386 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 509568000 bytes 2024-09-11T09:13:15.566 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 556417024 bytes 2024-09-11T09:13:15.732 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 595836928 bytes 2024-09-11T09:13:15.905 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 635208704 bytes 2024-09-11T09:13:16.048 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 652362752 bytes 2024-09-11T09:13:16.266 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 727092224 bytes 2024-09-11T09:13:16.573 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 822141952 bytes 2024-09-11T09:13:16.674 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 822753280 bytes 2024-09-11T09:13:16.869 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 870824960 bytes 2024-09-11T09:13:17.077 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 922717184 bytes 2024-09-11T09:13:17.322 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 997702656 bytes 2024-09-11T09:13:17.598 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1095282688 bytes 2024-09-11T09:13:17.735 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1116039168 bytes 2024-09-11T09:13:17.886 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1147275264 bytes 2024-09-11T09:13:18.041 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1176054784 bytes 2024-09-11T09:13:18.222 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1220881408 bytes 2024-09-11T09:13:18.371 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1245552640 bytes 2024-09-11T09:13:18.522 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1270537216 bytes 2024-09-11T09:13:18.780 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1358529536 bytes 2024-09-11T09:13:18.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1384977408 bytes 2024-09-11T09:13:19.026 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1392142336 bytes 2024-09-11T09:13:19.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1431159808 bytes 2024-09-11T09:13:19.391 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1490225152 bytes 2024-09-11T09:13:19.582 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1531271168 bytes 2024-09-11T09:13:19.733 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1559250944 bytes 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_8 present 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:13:19.934 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_8 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T09:13:19.935 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:13:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a1c830403c 2024-09-11T09:13:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:13:19.987 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:13:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_8 present 12a1c830403c 2024-09-11T09:13:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:13:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:13:20.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a1c830403c 2024-09-11T09:13:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:13:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:13:20.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T09:13:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:13:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:13:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_8 2024-09-11T09:13:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T09:13:20.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a1c830403c 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a1c830403c 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a1c830403c -o 12a1c830403c = 12a1c830403c 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_8 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:20.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:20.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_8 2024-09-11T09:13:20.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:13:20.985+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3b9f3000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=192 2024-09-11T09:13:20.987 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:13:20.985+0000 7febbbf1a640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3b9f3000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=194, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=c0, last_copied_object_number=128, snap_seqs={192=18446744073709551614}] 2024-09-11T09:13:24.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.0 with skip (0.7101407124201098) > weight (0.25) 2024-09-11T09:13:24.540 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster1.client.mirror.1 with skip (0.311606672063677) > weight (0.25) 2024-09-11T09:13:24.540 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.2 2024-09-11T09:13:24.540 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Sent signal 15 2024-09-11T09:13:24.541 INFO:tasks.rbd_mirror_thrash:waiting for 7 secs before reviving daemons 2024-09-11T09:13:24.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:13:24.541+0000 7ff1f60de640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 58260) UID: 1000 2024-09-11T09:13:25.977 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_8 deleted 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:13:25.992 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:13:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T09:13:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:13:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:13:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_8 2024-09-11T09:13:25.993 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T09:13:25.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:13:26.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T09:13:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:13:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:13:26.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_8 deleted '' 2024-09-11T09:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:13:26.352 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_8 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_8 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_8 2024-09-11T09:13:26.353 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:13:26.398 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_8: (2) No such file or directory 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_8 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:26.403 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_8 2024-09-11T09:13:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_8 2024-09-11T09:13:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:13:26.452 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:13:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_8 2024-09-11T09:13:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_8 2024-09-11T09:13:26.453 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_8 2024-09-11T09:13:26.500 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_8 2024-09-11T09:13:31.541 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.2 2024-09-11T09:13:31.541 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:13:31.542 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.2 2024-09-11T09:13:31.542 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Restarting daemon 2024-09-11T09:13:31.542 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 2024-09-11T09:13:31.545 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Started 2024-09-11T09:13:31.545 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-11T09:13:49.381 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:13:49.385 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_9 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_9 512M 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:49.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:13:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:13:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:13:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:13:49.389 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_9 2024-09-11T09:13:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_9 2024-09-11T09:13:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:13:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:13:49.436 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_9 snapshot 2024-09-11T09:13:49.845 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:13:49.861 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_9 2024-09-11T09:13:49.906 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_9': 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 19f99100e305 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.19f99100e305 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:13:49 2024 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:13:49 2024 2024-09-11T09:13:49.907 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:13:49 2024 2024-09-11T09:13:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:13:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:13:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: 5dfbbeb5-8519-4ce8-a26b-03984cf4bf4f 2024-09-11T09:13:49.908 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:13:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_9 2024-09-11T09:13:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:13:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:13:49.911 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:13:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=10 2024-09-11T09:13:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:13:49.913 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 10s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_9 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:13:50.129 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 39944192 bytes 2024-09-11T09:13:50.276 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 65971200 bytes 2024-09-11T09:13:50.416 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 96334848 bytes 2024-09-11T09:13:50.603 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 136465408 bytes 2024-09-11T09:13:50.799 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 187676672 bytes 2024-09-11T09:13:50.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 200500224 bytes 2024-09-11T09:13:51.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 201103360 bytes 2024-09-11T09:13:51.183 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 244582400 bytes 2024-09-11T09:13:51.310 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 269109248 bytes 2024-09-11T09:13:51.409 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 270148608 bytes 2024-09-11T09:13:51.614 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 330536960 bytes 2024-09-11T09:13:51.716 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 338334720 bytes 2024-09-11T09:13:51.816 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 342721536 bytes 2024-09-11T09:13:51.976 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 376330240 bytes 2024-09-11T09:13:52.226 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 457072640 bytes 2024-09-11T09:13:52.366 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 481558528 bytes 2024-09-11T09:13:52.627 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 572777472 bytes 2024-09-11T09:13:52.751 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 593735680 bytes 2024-09-11T09:13:52.994 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 663910400 bytes 2024-09-11T09:13:53.181 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 708205568 bytes 2024-09-11T09:13:53.354 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 751591424 bytes 2024-09-11T09:13:53.519 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 784400384 bytes 2024-09-11T09:13:53.749 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 857164800 bytes 2024-09-11T09:13:53.912 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 895631360 bytes 2024-09-11T09:13:54.009 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 896237568 bytes 2024-09-11T09:13:54.153 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 922903552 bytes 2024-09-11T09:13:54.370 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 991963136 bytes 2024-09-11T09:13:54.579 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1053631488 bytes 2024-09-11T09:13:54.851 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1148014592 bytes 2024-09-11T09:13:55.021 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1182323712 bytes 2024-09-11T09:13:55.160 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1209003008 bytes 2024-09-11T09:13:55.294 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1231044608 bytes 2024-09-11T09:13:55.456 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1270488064 bytes 2024-09-11T09:13:55.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1303786496 bytes 2024-09-11T09:13:55.795 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1343672320 bytes 2024-09-11T09:13:55.964 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1404516352 bytes 2024-09-11T09:13:56.176 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1442967552 bytes 2024-09-11T09:13:56.273 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 1443562496 bytes 2024-09-11T09:13:56.506 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 1519130624 bytes 2024-09-11T09:13:56.707 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 1572335616 bytes 2024-09-11T09:13:56.938 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4100 IOs, 1650921472 bytes 2024-09-11T09:13:57.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4200 IOs, 1701939200 bytes 2024-09-11T09:13:57.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4300 IOs, 1725372416 bytes 2024-09-11T09:13:57.397 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4400 IOs, 1729622016 bytes 2024-09-11T09:13:57.568 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4500 IOs, 1770013696 bytes 2024-09-11T09:13:57.796 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4600 IOs, 1838136320 bytes 2024-09-11T09:13:58.054 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4700 IOs, 1899407360 bytes 2024-09-11T09:13:58.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4800 IOs, 1949905920 bytes 2024-09-11T09:13:58.643 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4900 IOs, 2004201472 bytes 2024-09-11T09:13:58.811 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5000 IOs, 2035298304 bytes 2024-09-11T09:13:58.991 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5100 IOs, 2073415680 bytes 2024-09-11T09:13:59.245 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5200 IOs, 2155735040 bytes 2024-09-11T09:13:59.468 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5300 IOs, 2230515712 bytes 2024-09-11T09:13:59.698 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5400 IOs, 2276492288 bytes 2024-09-11T09:13:59.817 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:5500 IOs, 2288480256 bytes 2024-09-11T09:13:59.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_9 present 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:13:59.922 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:13:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T09:13:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:13:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:13:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_9 2024-09-11T09:13:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T09:13:59.923 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:13:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a1467062ea 2024-09-11T09:13:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:13:59.980 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:14:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_9 present 12a1467062ea 2024-09-11T09:14:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:14:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:14:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:14:00.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a1467062ea 2024-09-11T09:14:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:14:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:14:00.082 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T09:14:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:14:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:14:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_9 2024-09-11T09:14:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T09:14:00.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a1467062ea 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a1467062ea 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a1467062ea -o 12a1467062ea = 12a1467062ea 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_9 2024-09-11T09:14:00.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:14:00.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_9 2024-09-11T09:14:00.997 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:14:01.000+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3e7a3000 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=193 2024-09-11T09:14:00.998 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:14:01.000+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3e7a3000 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=195, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=c1, last_copied_object_number=128, snap_seqs={193=18446744073709551614}] 2024-09-11T09:14:04.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:14:04.719+0000 7fd6851e5640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece4976340 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:14:04.719 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:14:04.719+0000 7fd68b1f1640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3aa00 [3/5dfbbeb5-8519-4ce8-a26b-03984cf4bf4f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:14:05.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:14:05.519+0000 7fd6849e4640 -1 rbd::mirror::image_replayer::BootstrapRequest: 0x55ece5147ba0 handle_prepare_local_image: error preparing local image for replay: (85) Interrupted system call should be restarted 2024-09-11T09:14:05.518 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:14:05.519+0000 7fd68b9f2640 -1 rbd::mirror::ImageReplayer: 0x55ece4c3aa00 [3/5dfbbeb5-8519-4ce8-a26b-03984cf4bf4f] operator(): start failed: (85) Interrupted system call should be restarted 2024-09-11T09:14:05.988 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:14:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_9 deleted 2024-09-11T09:14:06.000 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:14:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:14:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:14:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:14:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:14:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:14:06.001 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T09:14:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:14:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:14:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_9 2024-09-11T09:14:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T09:14:06.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:14:06.045 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T09:14:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:14:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:14:06.049 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:14:06.150 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_9 deleted '' 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_9 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:14:06.151 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:14:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_9 2024-09-11T09:14:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_9 2024-09-11T09:14:06.152 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:14:06.197 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_9: (2) No such file or directory 2024-09-11T09:14:06.201 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_9 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:14:06.202 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_9 2024-09-11T09:14:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_9 2024-09-11T09:14:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:14:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:14:06.250 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:14:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_9 2024-09-11T09:14:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_9 2024-09-11T09:14:06.251 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_9 2024-09-11T09:14:06.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_9 2024-09-11T09:14:17.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.0 with skip (0.791060238722089) > weight (0.25) 2024-09-11T09:14:17.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.1 with skip (0.7361417333758574) > weight (0.25) 2024-09-11T09:14:17.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.2 with skip (0.2504705523833447) > weight (0.25) 2024-09-11T09:14:17.435 INFO:tasks.rbd_mirror_thrash:skipping daemon cluster2.client.mirror.3 with skip (0.8322695858751568) > weight (0.25) 2024-09-11T09:14:17.436 INFO:tasks.rbd_mirror_thrash:waiting for 89 secs before thrashing 2024-09-11T09:14:29.383 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for i in `seq 1 10` 2024-09-11T09:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image=image_10 2024-09-11T09:14:29.387 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ create_image cluster2 mirror image_10 512M 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local size=128 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -n 512M ']' 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ size=512M 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ shift 2024-09-11T09:14:29.388 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 create --size 512M --image-feature layering,exclusive-lock mirror/image_10 2024-09-11T09:14:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ enable_mirror cluster2 mirror image_10 2024-09-11T09:14:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local mode=snapshot 2024-09-11T09:14:29.437 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image enable mirror/image_10 snapshot 2024-09-11T09:14:29.848 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring enabled 2024-09-11T09:14:29.865 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 info mirror/image_10 2024-09-11T09:14:29.914 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:rbd image 'image_10': 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: size 512 MiB in 128 objects 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: order 22 (4 MiB objects) 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: snapshot_count: 1 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: id: 1a11e66e2803 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: block_name_prefix: rbd_data.1a11e66e2803 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: format: 2 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: features: layering, exclusive-lock 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: op_features: 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: flags: 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: create_timestamp: Wed Sep 11 09:14:29 2024 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: access_timestamp: Wed Sep 11 09:14:29 2024 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: modify_timestamp: Wed Sep 11 09:14:29 2024 2024-09-11T09:14:29.915 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring state: enabled 2024-09-11T09:14:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring mode: snapshot 2024-09-11T09:14:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring global id: d3458379-d248-4252-8026-fa03f75cbece 2024-09-11T09:14:29.916 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout: mirroring primary: true 2024-09-11T09:14:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stress_write_image cluster2 mirror image_10 2024-09-11T09:14:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:29.919 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ awk 'BEGIN {srand(); print int(10 * rand()) + 5}' 2024-09-11T09:14:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local duration=8 2024-09-11T09:14:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:14:29.921 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ timeout 8s ceph_test_rbd_mirror_random_write --cluster cluster2 mirror image_10 --debug-rbd=20 --debug-journaler=20 2024-09-11T09:14:30.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:100 IOs, 35808256 bytes 2024-09-11T09:14:30.440 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:200 IOs, 146467840 bytes 2024-09-11T09:14:30.558 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:300 IOs, 168627200 bytes 2024-09-11T09:14:30.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:400 IOs, 255988736 bytes 2024-09-11T09:14:31.002 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:500 IOs, 304030720 bytes 2024-09-11T09:14:31.208 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:600 IOs, 367457280 bytes 2024-09-11T09:14:31.380 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:700 IOs, 404685824 bytes 2024-09-11T09:14:31.523 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:800 IOs, 429335552 bytes 2024-09-11T09:14:31.645 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:900 IOs, 449253376 bytes 2024-09-11T09:14:31.808 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1000 IOs, 488111104 bytes 2024-09-11T09:14:31.996 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1100 IOs, 531183616 bytes 2024-09-11T09:14:32.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1200 IOs, 548472832 bytes 2024-09-11T09:14:32.405 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1300 IOs, 658320384 bytes 2024-09-11T09:14:32.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1400 IOs, 746895360 bytes 2024-09-11T09:14:32.843 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1500 IOs, 788227072 bytes 2024-09-11T09:14:33.038 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1600 IOs, 835586048 bytes 2024-09-11T09:14:33.260 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1700 IOs, 902813696 bytes 2024-09-11T09:14:33.459 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1800 IOs, 959112192 bytes 2024-09-11T09:14:33.630 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:1900 IOs, 1002045440 bytes 2024-09-11T09:14:33.829 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2000 IOs, 1049809920 bytes 2024-09-11T09:14:34.137 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2100 IOs, 1158224896 bytes 2024-09-11T09:14:34.432 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2200 IOs, 1251704832 bytes 2024-09-11T09:14:34.587 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2300 IOs, 1283976192 bytes 2024-09-11T09:14:34.782 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2400 IOs, 1348719616 bytes 2024-09-11T09:14:35.007 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2500 IOs, 1417115648 bytes 2024-09-11T09:14:35.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2600 IOs, 1466094592 bytes 2024-09-11T09:14:35.357 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2700 IOs, 1507050496 bytes 2024-09-11T09:14:35.477 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2800 IOs, 1520617472 bytes 2024-09-11T09:14:35.690 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:2900 IOs, 1575251968 bytes 2024-09-11T09:14:35.858 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3000 IOs, 1609672704 bytes 2024-09-11T09:14:35.967 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3100 IOs, 1618652160 bytes 2024-09-11T09:14:36.184 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3200 IOs, 1688892416 bytes 2024-09-11T09:14:36.372 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3300 IOs, 1725993984 bytes 2024-09-11T09:14:36.498 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3400 IOs, 1744566272 bytes 2024-09-11T09:14:36.678 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3500 IOs, 1789689856 bytes 2024-09-11T09:14:36.803 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3600 IOs, 1806369792 bytes 2024-09-11T09:14:37.098 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3700 IOs, 1910104064 bytes 2024-09-11T09:14:37.395 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3800 IOs, 2030430208 bytes 2024-09-11T09:14:37.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:3900 IOs, 2085045248 bytes 2024-09-11T09:14:37.819 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:4000 IOs, 2136291328 bytes 2024-09-11T09:14:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=124 2024-09-11T09:14:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:14:37.930 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 124 -eq 124 ']' 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_10 present 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=present 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:14:37.931 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_10 2024-09-11T09:14:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T09:14:37.932 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:14:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id=12a1e4827a53 2024-09-11T09:14:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:14:37.981 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_10 present 12a1e4827a53 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=present 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id=12a1e4827a53 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:14:38.083 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:14:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T09:14:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:14:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:14:38.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_10 2024-09-11T09:14:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T09:14:38.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:14:38.142 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id=12a1e4827a53 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 12a1e4827a53 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -z 12a1e4827a53 -o 12a1e4827a53 = 12a1e4827a53 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_state=present 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test present = present 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ disable_mirror cluster2 mirror image_10 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:38.143 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 mirror image disable mirror/image_10 2024-09-11T09:14:38.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:14:38.947+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3b9f2400 scan_remote_mirror_snapshots: failed to locate remote start snapshot: snap_id=194 2024-09-11T09:14:38.950 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:14:38.947+0000 7febbc71b640 -1 rbd::mirror::image_replayer::snapshot::Replayer: 0x560f3b9f2400 scan_remote_mirror_snapshots: split-brain detected: failed to find matching non-primary snapshot in remote image: local_snap_id_start=196, local_snap_ns=[mirror state=non-primary, complete=1, mirror_peer_uuids=, primary_mirror_uuid=16cd46a8-7a95-494f-a9ad-bfcebd7166e0, primary_snap_id=c2, last_copied_object_number=128, snap_seqs={194=18446744073709551614}] 2024-09-11T09:14:43.948 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:Mirroring disabled 2024-09-11T09:14:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_present cluster1 mirror image_10 deleted 2024-09-11T09:14:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:14:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local state=deleted 2024-09-11T09:14:44.003 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local s 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_10 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T09:14:44.004 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:14:44.014 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T09:14:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ image_id= 2024-09-11T09:14:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0.1 1 2 4 8 8 8 8 8 8 8 8 16 16 32 32 2024-09-11T09:14:44.018 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0.1 2024-09-11T09:14:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test_image_present cluster1 mirror image_10 deleted '' 2024-09-11T09:14:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:14:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local test_state=deleted 2024-09-11T09:14:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image_id= 2024-09-11T09:14:44.120 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_state=deleted 2024-09-11T09:14:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local current_image_id 2024-09-11T09:14:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ get_image_id cluster1 mirror image_10 2024-09-11T09:14:44.121 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:14:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:14:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local image=image_10 2024-09-11T09:14:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rbd --cluster=cluster1 info mirror/image_10 2024-09-11T09:14:44.122 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ sed -ne 's/^.*block_name_prefix: rbd_data\.//p' 2024-09-11T09:14:44.190 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:rbd: error opening image image_10: (2) No such file or directory 2024-09-11T09:14:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ current_image_id= 2024-09-11T09:14:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n '' 2024-09-11T09:14:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test deleted = deleted 2024-09-11T09:14:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:14:44.195 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ purge_snapshots cluster2 mirror image_10 2024-09-11T09:14:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:44.196 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster cluster2 snap purge mirror/image_10 2024-09-11T09:14:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image_retry cluster2 mirror image_10 2024-09-11T09:14:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 4 8 16 32 2024-09-11T09:14:44.246 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:14:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ remove_image cluster2 mirror image_10 2024-09-11T09:14:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:14:44.247 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local image=image_10 2024-09-11T09:14:44.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 snap purge mirror/image_10 2024-09-11T09:14:44.297 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rbd --cluster=cluster2 rm mirror/image_10 2024-09-11T09:15:00.546 INFO:tasks.rbd_mirror_thrash:kill cluster1.client.mirror.0 2024-09-11T09:15:00.547 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Sent signal 15 2024-09-11T09:15:00.547 INFO:tasks.rbd_mirror_thrash:waiting for 3 secs before reviving daemons 2024-09-11T09:15:00.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:00.550+0000 7f54c22f5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 64901) UID: 1000 2024-09-11T09:15:03.548 INFO:tasks.rbd_mirror_thrash:waiting for cluster1.client.mirror.0 2024-09-11T09:15:03.548 INFO:teuthology.orchestra.run:waiting for 600 2024-09-11T09:15:03.548 INFO:tasks.rbd_mirror_thrash:reviving cluster1.client.mirror.0 2024-09-11T09:15:03.548 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Restarting daemon 2024-09-11T09:15:03.549 DEBUG:teuthology.orchestra.run.smithi077:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 2024-09-11T09:15:03.552 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Started 2024-09-11T09:15:03.552 INFO:tasks.rbd_mirror_thrash:waiting for 109 secs before thrashing 2024-09-11T09:15:07.529 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr: Removing image: 1% complete... Removing image: 2% complete... Removing image: 3% complete... Removing image: 4% complete... Removing image: 5% complete... Removing image: 6% complete... Removing image: 7% complete... Removing image: 8% complete... Removing image: 9% complete... Removing image: 10% complete... Removing image: 11% complete... Removing image: 12% complete... Removing image: 13% complete... Removing image: 14% complete... Removing image: 15% complete... Removing image: 16% complete... Removing image: 17% complete... Removing image: 18% complete... Removing image: 19% complete... Removing image: 20% complete... Removing image: 21% complete... Removing image: 22% complete... Removing image: 23% complete... Removing image: 24% complete... Removing image: 25% complete... Removing image: 26% complete... Removing image: 27% complete... Removing image: 28% complete... Removing image: 29% complete... Removing image: 30% complete... Removing image: 31% complete... Removing image: 32% complete... Removing image: 33% complete... Removing image: 34% complete... Removing image: 35% complete... Removing image: 36% complete... Removing image: 37% complete... Removing image: 38% complete... Removing image: 39% complete... Removing image: 40% complete... Removing image: 41% complete... Removing image: 42% complete... Removing image: 43% complete... Removing image: 44% complete... Removing image: 45% complete... Removing image: 46% complete... Removing image: 47% complete... Removing image: 48% complete... Removing image: 49% complete... Removing image: 50% complete... Removing image: 51% complete... Removing image: 52% complete... Removing image: 53% complete... Removing image: 54% complete... Removing image: 55% complete... Removing image: 56% complete... Removing image: 57% complete... Removing image: 58% complete... Removing image: 59% complete... Removing image: 60% complete... Removing image: 61% complete... Removing image: 62% complete... Removing image: 63% complete... Removing image: 64% complete... Removing image: 65% complete... Removing image: 66% complete... Removing image: 67% complete... Removing image: 68% complete... Removing image: 69% complete... Removing image: 70% complete... Removing image: 71% complete... Removing image: 72% complete... Removing image: 73% complete... Removing image: 74% complete... Removing image: 75% complete... Removing image: 76% complete... Removing image: 77% complete... Removing image: 78% complete... Removing image: 79% complete... Removing image: 80% complete... Removing image: 81% complete... Removing image: 82% complete... Removing image: 83% complete... Removing image: 84% complete... Removing image: 85% complete... Removing image: 86% complete... Removing image: 87% complete... Removing image: 88% complete... Removing image: 89% complete... Removing image: 90% complete... Removing image: 91% complete... Removing image: 92% complete... Removing image: 93% complete... Removing image: 94% complete... Removing image: 95% complete... Removing image: 96% complete... Removing image: 97% complete... Removing image: 98% complete... Removing image: 99% complete... Removing image: 100% complete...done. 2024-09-11T09:15:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:15:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ testlog 'TEST: check if removed images'\'' OMAP are removed' 2024-09-11T09:15:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ tee -a /tmp/tmp.LFCa6yxq8J/rbd-mirror.test.log 2024-09-11T09:15:07.533 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ date '+%F %T' 2024-09-11T09:15:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo 2024-09-11 09:15:07 TEST: check if removed 'images'\''' OMAP are removed 2024-09-11T09:15:07.534 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:2024-09-11 09:15:07 TEST: check if removed images' OMAP are removed 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_in_omap cluster1 mirror 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring status_global 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local obj_name=rbd_mirroring 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local filter=status_global 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T09:15:07.535 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:15:07.536 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring status_global 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local filter=status_global 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -c status_global 2024-09-11T09:15:07.537 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 = 0 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirroring image_ 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local obj_name=rbd_mirroring 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local filter=image_ 2024-09-11T09:15:07.575 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T09:15:07.576 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirroring image_ 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local filter=image_ 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ list_omap_keys cluster1 mirror rbd_mirroring 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -c image_ 2024-09-11T09:15:07.577 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirroring 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 = 0 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_omap_keys cluster1 mirror rbd_mirror_leader image_map 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:15:07.616 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local obj_name=rbd_mirror_leader 2024-09-11T09:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local filter=image_map 2024-09-11T09:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T09:15:07.617 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:15:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:15:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ count_omap_keys_with_filter cluster1 mirror rbd_mirror_leader image_map 2024-09-11T09:15:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:15:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T09:15:07.618 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local filter=image_map 2024-09-11T09:15:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ list_omap_keys cluster1 mirror rbd_mirror_leader 2024-09-11T09:15:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster1 2024-09-11T09:15:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -c image_map 2024-09-11T09:15:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T09:15:07.619 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rados --cluster cluster1 -p mirror listomapkeys rbd_mirror_leader 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 = 0 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_image_in_omap cluster2 mirror 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring status_global 2024-09-11T09:15:07.657 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:15:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local obj_name=rbd_mirroring 2024-09-11T09:15:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local filter=status_global 2024-09-11T09:15:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T09:15:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:15:07.658 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:15:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring status_global 2024-09-11T09:15:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster2 2024-09-11T09:15:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local filter=status_global 2024-09-11T09:15:07.659 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-11T09:15:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster2 2024-09-11T09:15:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-11T09:15:07.660 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -c status_global 2024-09-11T09:15:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 = 0 2024-09-11T09:15:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T09:15:07.692 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirroring image_ 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local obj_name=rbd_mirroring 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local filter=image_ 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T09:15:07.693 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:15:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:15:07.694 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirroring image_ 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster2 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local filter=image_ 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ list_omap_keys cluster2 mirror rbd_mirroring 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster2 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -c image_ 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirroring 2024-09-11T09:15:07.695 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirroring 2024-09-11T09:15:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 = 0 2024-09-11T09:15:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T09:15:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:15:07.728 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ wait_for_omap_keys cluster2 mirror rbd_mirror_leader image_map 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local pool=mirror 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local obj_name=rbd_mirror_leader 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local filter=image_map 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for s in 0 1 2 2 4 4 8 8 8 16 16 32 2024-09-11T09:15:07.729 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ sleep 0 2024-09-11T09:15:07.730 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ count_omap_keys_with_filter cluster2 mirror rbd_mirror_leader image_map 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster2 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local filter=image_map 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ list_omap_keys cluster2 mirror rbd_mirror_leader 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local cluster=cluster2 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local pool=mirror 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ local obj_name=rbd_mirror_leader 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ grep -c image_map 2024-09-11T09:15:07.731 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ rados --cluster cluster2 -p mirror listomapkeys rbd_mirror_leader 2024-09-11T09:15:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test 0 = 0 2024-09-11T09:15:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ error_code=0 2024-09-11T09:15:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set -e 2024-09-11T09:15:07.764 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 0 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ cleanup 0 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local error_code=0 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ set +e 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -ne 0 ']' 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -z '' ']' 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster instance 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T09:15:07.765 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster1 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-11T09:15:08.207 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror' does not exist 2024-09-11T09:15:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T09:15:08.224 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster2 osd pool rm mirror mirror --yes-i-really-really-mean-it 2024-09-11T09:15:09.099 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror' does not exist 2024-09-11T09:15:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T09:15:09.125 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster1 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-11T09:15:09.624 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.626+0000 7f7d0179e640 -1 rbd::mirror::RemotePoolPoller: 0x55e1e2bd20d0 handle_mirror_peer_ping: failed to ping remote mirror peer: (2) No such file or directory 2024-09-11T09:15:09.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.626+0000 7f7d01f9f640 -1 rbd::mirror::RemotePoolPoller: 0x55e1e2bd20d0 handle_mirror_peer_list: failed to retrieve mirror peers: (2) No such file or directory 2024-09-11T09:15:09.625 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.626+0000 7f7d01f9f640 -1 rbd::mirror::PoolReplayer: 0x55e1e2be3b00 handle_remote_pool_meta_updated: remote pool metadata updated unexpectedly 2024-09-11T09:15:09.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.626+0000 7f7d0e7b8640 -1 rbd::mirror::InstanceWatcher: 0x55e1e3d20700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5714 object: (2) No such file or directory 2024-09-11T09:15:09.626 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.626+0000 7f7d0e7b8640 -1 rbd::mirror::InstanceWatcher: 0x55e1e3d20380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5714 object: (2) No such file or directory 2024-09-11T09:15:09.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.626+0000 7f7d087ac640 -1 rbd::mirror::InstanceWatcher: 0x55e1e3d20700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:09.627 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.626+0000 7f7d07fab640 -1 rbd::mirror::InstanceWatcher: 0x55e1e3d20380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:09.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d0179e640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e3d626c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:09.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d0179e640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e3d62360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:09.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d0e7b8640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3d601a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:09.628 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d0e7b8640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3d60000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:09.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d087ac640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e3d62120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:09.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d07fab640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e3d62000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:09.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d0efb9640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3d601a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:09.629 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.630+0000 7f7d0e7b8640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3d60000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:09.671 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.670+0000 7fd68b1f1640 -1 rbd::mirror::LeaderWatcher: 0x55ece5506c00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-11T09:15:09.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd6851e5640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ece507fd40 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T09:15:09.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd6851e5640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ece507fb00 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T09:15:09.672 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd6851e5640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55ece507f440 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T09:15:09.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd6849e4640 -1 rbd::mirror::PoolWatcher: 0x55ece564c780 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:09.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd6849e4640 -1 rbd::mirror::PoolWatcher: 0x55ece5095a40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:09.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd6849e4640 -1 rbd::mirror::PoolWatcher: 0x55ece564c5a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:09.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd67e1d7640 -1 rbd::mirror::PoolWatcher: 0x55ece509f680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:09.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd67e1d7640 -1 rbd::mirror::PoolWatcher: 0x55ece4e5dc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:09.673 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd67e1d7640 -1 rbd::mirror::PoolWatcher: 0x55ece483ba40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:09.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd68b9f2640 -1 rbd::mirror::LeaderWatcher: 0x55ece5506c00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T09:15:09.674 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:09.674+0000 7fd68b9f2640 -1 rbd::mirror::LeaderWatcher: 0x55ece5506c00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-11T09:15:09.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.830+0000 7f7d0e7b8640 -1 rbd::mirror::LeaderWatcher: 0x55e1e3c6ec00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:09.831 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.830+0000 7f7d0efb9640 -1 rbd::mirror::InstanceWatcher: 0x55e1e3d20000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5714 object: (2) No such file or directory 2024-09-11T09:15:09.832 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.834+0000 7f7d087ac640 -1 rbd::mirror::InstanceWatcher: 0x55e1e3d20000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:09.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.834+0000 7f7d01f9f640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e47726c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:09.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.834+0000 7f7d0efb9640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3a8c340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:09.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.834+0000 7f7d07fab640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e47170e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:09.833 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:09.834+0000 7f7d0e7b8640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3a8c340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:10.133 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.134+0000 7febc9735640 -1 rbd::mirror::LeaderWatcher: 0x560f3b7fac00 handle_notify_heartbeat: error notifying heartbeat: (2) No such file or directory, releasing leader 2024-09-11T09:15:10.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.134+0000 7febc2f28640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560f3c030480 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T09:15:10.135 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.134+0000 7febc2f28640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560f3bf41680 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T09:15:10.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.134+0000 7febc2727640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x560f3c030240 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T09:15:10.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febc2f28640 -1 rbd::mirror::PoolWatcher: 0x560f3aedbc20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:10.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febc2f28640 -1 rbd::mirror::PoolWatcher: 0x560f3aedbe00 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:10.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febbc71b640 -1 rbd::mirror::PoolWatcher: 0x560f3aeda1e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:10.136 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febbc71b640 -1 rbd::mirror::PoolWatcher: 0x560f3aedb680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:10.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febbbf1a640 -1 rbd::mirror::PoolWatcher: 0x560f3aeda5a0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:10.137 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febc2727640 -1 rbd::mirror::PoolWatcher: 0x560f3eaab680 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:10.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febc9735640 -1 rbd::mirror::LeaderWatcher: 0x560f3b7fac00 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T09:15:10.138 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:10.138+0000 7febc9735640 -1 rbd::mirror::LeaderWatcher: 0x560f3b7fac00 handle_release_leader_lock: error releasing lock: (2) No such file or directory 2024-09-11T09:15:10.231 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror_parent' does not exist 2024-09-11T09:15:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ CEPH_ARGS= 2024-09-11T09:15:10.248 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ ceph --cluster cluster2 osd pool rm mirror_parent mirror_parent --yes-i-really-really-mean-it 2024-09-11T09:15:11.063 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:pool 'mirror_parent' does not exist 2024-09-11T09:15:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-11T09:15:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirrors cluster1 2024-09-11T09:15:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1 2024-09-11T09:15:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.081 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 0 3 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster1:0 '' 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:0 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster1:1 '' 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:1 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.084 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster1:2 '' 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:2 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster1:3 '' 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster1:3 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.085 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for cluster in "${CLUSTER1}" "${CLUSTER2}" 2024-09-11T09:15:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirrors cluster2 2024-09-11T09:15:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2 2024-09-11T09:15:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.086 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:++ seq 0 3 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster2:0 '' 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:0 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster2:1 '' 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:1 2024-09-11T09:15:11.087 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster2:2 '' 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:2 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ for instance in `seq 0 ${LAST_MIRROR_INSTANCE}` 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ stop_mirror cluster2:3 '' 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local cluster=cluster2:3 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ local sig= 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test -n 1 2024-09-11T09:15:11.088 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ return 2024-09-11T09:15:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' -z 1 ']' 2024-09-11T09:15:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ test '' = /tmp/tmp.LFCa6yxq8J 2024-09-11T09:15:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ rm -Rf /tmp/tmp.LFCa6yxq8J 2024-09-11T09:15:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ '[' 0 -eq 0 ']' 2024-09-11T09:15:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ echo OK 2024-09-11T09:15:11.089 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stdout:OK 2024-09-11T09:15:11.090 INFO:tasks.workunit.cluster1.client.mirror.smithi077.stderr:+ exit 0 2024-09-11T09:15:11.090 INFO:teuthology.orchestra.run:Running command with timeout 3600 2024-09-11T09:15:11.091 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror/tmp 2024-09-11T09:15:11.145 INFO:tasks.workunit:Stopping ['rbd/rbd_mirror_stress.sh'] on cluster1.client.mirror... 2024-09-11T09:15:11.146 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/workunits.list.cluster1.client.mirror /home/ubuntu/cephtest/clone.cluster1.client.mirror 2024-09-11T09:15:11.521 DEBUG:teuthology.parallel:result is None 2024-09-11T09:15:11.522 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T09:15:11.533 INFO:tasks.workunit:Deleted dir /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T09:15:11.533 DEBUG:teuthology.orchestra.run.smithi077:> rmdir -- /home/ubuntu/cephtest/mnt.cluster1.mirror 2024-09-11T09:15:11.653 INFO:tasks.workunit:Deleted artificial mount point /home/ubuntu/cephtest/mnt.cluster1.mirror/client.mirror 2024-09-11T09:15:11.653 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-11T09:15:11.664 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-11T09:15:11.664 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 16 2024-09-11T09:15:11.665 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-11T09:15:11.665 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror-thrash 2024-09-11T09:15:11.828 INFO:tasks.rbd_mirror_thrash:joining rbd_mirror_thrash 2024-09-11T09:15:11.828 INFO:tasks.rbd_mirror_thrash:stat['kill'] = 20 2024-09-11T09:15:11.828 INFO:tasks.rbd_mirror_thrash:done joining 2024-09-11T09:15:11.829 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:11.858 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.3:waiting for process to exit 2024-09-11T09:15:11.859 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:11.859 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.858+0000 7f7d0f7ba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.3 (PID: 51329) UID: 1000 2024-09-11T09:15:11.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.862+0000 7f7cfaf91640 -1 rbd::mirror::image_deleter::TrashWatcher: 0x55e1e3d8e000 handle_unregister_watcher: error unregistering watcher for trash directory: (2) No such file or directory 2024-09-11T09:15:11.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.862+0000 7f7cfa790640 -1 rbd::mirror::PoolWatcher: 0x55e1e43ccb40 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:11.861 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.862+0000 7f7cf3f83640 -1 rbd::mirror::PoolWatcher: 0x55e1e3d99c20 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:11.862 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.862+0000 7f7d0efb9640 -1 rbd::mirror::LeaderWatcher: 0x55e1e3f1f200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T09:15:11.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.862+0000 7f7d0efb9640 -1 rbd::mirror::LeaderWatcher: 0x55e1e3f1f200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-11T09:15:11.863 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.862+0000 7f7d0e7b8640 -1 rbd::mirror::LeaderWatcher: 0x55e1e3f1f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:11.864 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.866+0000 7f7d0e7b8640 -1 rbd::mirror::InstanceWatcher: 0x55e1e41ca000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5716 object: (2) No such file or directory 2024-09-11T09:15:11.865 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.866+0000 7f7cfaf91640 -1 rbd::mirror::InstanceWatcher: 0x55e1e41ca000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:11.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.866+0000 7f7cf4784640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e3f5a360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.866+0000 7f7d0e7b8640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3a8c1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.866+0000 7f7cfa790640 -1 rbd::mirror::MirrorStatusUpdater 0x55e1e3f4f200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.866 INFO:tasks.rbd_mirror.cluster2.client.mirror.3.smithi077.stderr:2024-09-11T09:15:11.866+0000 7f7d0efb9640 -1 rbd::mirror::NamespaceReplayer: 0x55e1e3a8c1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.891 INFO:tasks.rbd_mirror.cluster2.client.mirror.3:Stopped 2024-09-11T09:15:11.891 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:11.931 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.2:waiting for process to exit 2024-09-11T09:15:11.931 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:11.932 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd68c1f3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.2 (PID: 42825) UID: 1000 2024-09-11T09:15:11.934 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd68b9f2640 -1 rbd::mirror::LeaderWatcher: 0x55ece492f200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:11.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd68b1f1640 -1 rbd::mirror::InstanceWatcher: 0x55ece471a700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5174 object: (2) No such file or directory 2024-09-11T09:15:11.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd68b1f1640 -1 rbd::mirror::InstanceWatcher: 0x55ece471a380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5174 object: (2) No such file or directory 2024-09-11T09:15:11.935 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd68b1f1640 -1 rbd::mirror::InstanceWatcher: 0x55ece4c24000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5177 object: (2) No such file or directory 2024-09-11T09:15:11.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd6851e5640 -1 rbd::mirror::InstanceWatcher: 0x55ece471a700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:11.936 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd6849e4640 -1 rbd::mirror::InstanceWatcher: 0x55ece471a380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:11.938 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd6771c9640 -1 rbd::mirror::InstanceWatcher: 0x55ece4c24000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:11.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd67e9d8640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece49665a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.934+0000 7fd67e9d8640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece4966120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd68b9f2640 -1 rbd::mirror::NamespaceReplayer: 0x55ece47a41a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd68b9f2640 -1 rbd::mirror::NamespaceReplayer: 0x55ece47a4000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.939 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd6851e5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece47a6120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd6851e5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece47a6000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd68b1f1640 -1 rbd::mirror::NamespaceReplayer: 0x55ece47a41a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd68b1f1640 -1 rbd::mirror::NamespaceReplayer: 0x55ece47a4000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd6711bd640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece4966b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.940 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd68b9f2640 -1 rbd::mirror::NamespaceReplayer: 0x55ece46921a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd6779ca640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece4960b40 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.938+0000 7fd68b1f1640 -1 rbd::mirror::NamespaceReplayer: 0x55ece46921a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.941 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.942+0000 7fd68b9f2640 -1 rbd::mirror::LeaderWatcher: 0x55ece5506c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:11.942 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.942+0000 7fd68b9f2640 -1 rbd::mirror::InstanceWatcher: 0x55ece471a000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5174 object: (2) No such file or directory 2024-09-11T09:15:11.943 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.942+0000 7fd6849e4640 -1 rbd::mirror::InstanceWatcher: 0x55ece471a000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:11.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.942+0000 7fd67e1d7640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece53806c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.942+0000 7fd68b1f1640 -1 rbd::mirror::NamespaceReplayer: 0x55ece4692340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.946+0000 7fd6851e5640 -1 rbd::mirror::MirrorStatusUpdater 0x55ece52df0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:11.944 INFO:tasks.rbd_mirror.cluster2.client.mirror.2.smithi077.stderr:2024-09-11T09:15:11.946+0000 7fd68b9f2640 -1 rbd::mirror::NamespaceReplayer: 0x55ece4692340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:11.976 INFO:tasks.rbd_mirror.cluster2.client.mirror.2:Stopped 2024-09-11T09:15:11.976 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:12.001 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.1:waiting for process to exit 2024-09-11T09:15:12.001 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:12.020 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f9445ca5640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.1 (PID: 64133) UID: 1000 2024-09-11T09:15:12.021 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f9444ca3640 -1 rbd::mirror::LeaderWatcher: 0x55939090b200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f943ec97640 -1 rbd::mirror::InstanceWatcher: 0x5593906fc380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f943ec97640 -1 rbd::mirror::InstanceWatcher: 0x5593906fca80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f94454a4640 -1 rbd::mirror::InstanceWatcher: 0x5593906fc700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6535 object: (2) No such file or directory 2024-09-11T09:15:12.023 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f943848a640 -1 rbd::mirror::MirrorStatusUpdater 0x5593907825a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f9444ca3640 -1 rbd::mirror::NamespaceReplayer: 0x5593907801a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f9437c89640 -1 rbd::mirror::MirrorStatusUpdater 0x559390782360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f943147c640 -1 rbd::mirror::InstanceWatcher: 0x5593906fc700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.022+0000 7f9444ca3640 -1 rbd::mirror::NamespaceReplayer: 0x559390780000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.026+0000 7f9430c7b640 -1 rbd::mirror::MirrorStatusUpdater 0x559390935200 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.024 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.026+0000 7f94454a4640 -1 rbd::mirror::NamespaceReplayer: 0x559390bb0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.029 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.030+0000 7f943e496640 -1 rbd::mirror::InstanceWatcher: 0x5593906fc000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.030+0000 7f943848a640 -1 rbd::mirror::MirrorStatusUpdater 0x5593911326c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.030 INFO:tasks.rbd_mirror.cluster2.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.030+0000 7f94454a4640 -1 rbd::mirror::NamespaceReplayer: 0x5593904461a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.053 INFO:tasks.rbd_mirror.cluster2.client.mirror.1:Stopped 2024-09-11T09:15:12.053 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:12.062 DEBUG:tasks.rbd_mirror.cluster2.client.mirror.0:waiting for process to exit 2024-09-11T09:15:12.063 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:12.100 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cb3e34640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster2 --id mirror.0 (PID: 63004) UID: 1000 2024-09-11T09:15:12.101 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cb2e32640 -1 rbd::mirror::LeaderWatcher: 0x55da44d5d200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cb2e32640 -1 rbd::mirror::InstanceWatcher: 0x55da44b75c00 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6457 object: (2) No such file or directory 2024-09-11T09:15:12.102 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cb3633640 -1 rbd::mirror::InstanceWatcher: 0x55da44b75880 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6457 object: (2) No such file or directory 2024-09-11T09:15:12.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cac625640 -1 rbd::mirror::InstanceWatcher: 0x55da44b75c00 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cb3633640 -1 rbd::mirror::InstanceWatcher: 0x55da45040000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6459 object: (2) No such file or directory 2024-09-11T09:15:12.103 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cace26640 -1 rbd::mirror::InstanceWatcher: 0x55da44b75880 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8ca6619640 -1 rbd::mirror::MirrorStatusUpdater 0x55da44d96360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8ca6619640 -1 rbd::mirror::MirrorStatusUpdater 0x55da44d965a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cb2e32640 -1 rbd::mirror::NamespaceReplayer: 0x55da44bf0000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8cb2e32640 -1 rbd::mirror::NamespaceReplayer: 0x55da44bf01a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.104 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.102+0000 7f8c9f60b640 -1 rbd::mirror::InstanceWatcher: 0x55da45040000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8cac625640 -1 rbd::mirror::MirrorStatusUpdater 0x55da44bf2000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8cb2e32640 -1 rbd::mirror::NamespaceReplayer: 0x55da44bf0000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8cace26640 -1 rbd::mirror::MirrorStatusUpdater 0x55da44bf2120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8cb2e32640 -1 rbd::mirror::NamespaceReplayer: 0x55da44bf01a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8c985fd640 -1 rbd::mirror::MirrorStatusUpdater 0x55da44d966c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8cb3633640 -1 rbd::mirror::NamespaceReplayer: 0x55da44bf04e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8c9ee0a640 -1 rbd::mirror::MirrorStatusUpdater 0x55da44d8ac60 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.105 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.106+0000 7f8cb2e32640 -1 rbd::mirror::NamespaceReplayer: 0x55da44bf04e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.110+0000 7f8cb2e32640 -1 rbd::mirror::LeaderWatcher: 0x55da44ad2c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.109 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.110+0000 7f8cb2e32640 -1 rbd::mirror::InstanceWatcher: 0x55da44b74000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6457 object: (2) No such file or directory 2024-09-11T09:15:12.110 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.110+0000 7f8cac625640 -1 rbd::mirror::InstanceWatcher: 0x55da44b74000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.110+0000 7f8ca5e18640 -1 rbd::mirror::MirrorStatusUpdater 0x55da449146c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.110+0000 7f8cb3633640 -1 rbd::mirror::NamespaceReplayer: 0x55da448ae340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.110+0000 7f8cace26640 -1 rbd::mirror::MirrorStatusUpdater 0x55da45575320 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.111 INFO:tasks.rbd_mirror.cluster2.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.110+0000 7f8cb2e32640 -1 rbd::mirror::NamespaceReplayer: 0x55da448ae340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.131 INFO:tasks.rbd_mirror.cluster2.client.mirror.0:Stopped 2024-09-11T09:15:12.131 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:12.141 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.3:waiting for process to exit 2024-09-11T09:15:12.141 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:12.172 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc9f36640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.3 (PID: 41874) UID: 1000 2024-09-11T09:15:12.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febae6ff640 -1 rbd::mirror::PoolWatcher: 0x560f3e7801e0 operator(): error unregistering watcher for rbd_mirroring object: (2) No such file or directory 2024-09-11T09:15:12.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc9735640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4769 object: (2) No such file or directory 2024-09-11T09:15:12.174 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc9735640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4769 object: (2) No such file or directory 2024-09-11T09:15:12.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc9735640 -1 rbd::mirror::LeaderWatcher: 0x560f3ac29200 handle_notify_lock_released: error notifying leader lock released: (2) No such file or directory 2024-09-11T09:15:12.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc9735640 -1 rbd::mirror::LeaderWatcher: 0x560f3ac29200 handle_shut_down_leader_lock: error shutting down leader lock: (2) No such file or directory 2024-09-11T09:15:12.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc2f28640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc2f28640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc9735640 -1 rbd::mirror::LeaderWatcher: 0x560f3ac29200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febbc71b640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3aa665a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.175 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febbc71b640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3aa66360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc8f34640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f84e0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.174+0000 7febc8f34640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f8680 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.176 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febc9735640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4772 object: (2) No such file or directory 2024-09-11T09:15:12.177 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febb4f0c640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febae6ff640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3aa666c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.178 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febc9735640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f81a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febb570d640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3ac050e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febc8f34640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f81a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febc2f28640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3aa66000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febc2f28640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3aa66120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febc9735640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f84e0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.179 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.178+0000 7febc9735640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f8680 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.183 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.182+0000 7febc9735640 -1 rbd::mirror::LeaderWatcher: 0x560f3b7fac00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.184 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.186+0000 7febc8f34640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.4769 object: (2) No such file or directory 2024-09-11T09:15:12.185 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.186+0000 7febc2f28640 -1 rbd::mirror::InstanceWatcher: 0x560f3aa04000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.186 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.186+0000 7febbbf1a640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3b65c6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.186+0000 7febc8f34640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f8000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.186+0000 7febc2727640 -1 rbd::mirror::MirrorStatusUpdater 0x560f3b5b0fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.187 INFO:tasks.rbd_mirror.cluster1.client.mirror.3.smithi077.stderr:2024-09-11T09:15:12.186+0000 7febc9735640 -1 rbd::mirror::NamespaceReplayer: 0x560f3a9f8000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.215 INFO:tasks.rbd_mirror.cluster1.client.mirror.3:Stopped 2024-09-11T09:15:12.215 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:12.225 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.2:waiting for process to exit 2024-09-11T09:15:12.226 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:12.256 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.258+0000 7f58733b3640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.2 (PID: 65669) UID: 1000 2024-09-11T09:15:12.258 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.258+0000 7f5872bb2640 -1 rbd::mirror::InstanceWatcher: 0x5632dc862a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6102 object: (2) No such file or directory 2024-09-11T09:15:12.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.258+0000 7f5872bb2640 -1 rbd::mirror::InstanceWatcher: 0x5632dc862380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6102 object: (2) No such file or directory 2024-09-11T09:15:12.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.258+0000 7f585e389640 -1 rbd::mirror::InstanceWatcher: 0x5632dcd22000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.258+0000 7f586bba4640 -1 rbd::mirror::InstanceWatcher: 0x5632dc862a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.259 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.258+0000 7f586bba4640 -1 rbd::mirror::InstanceWatcher: 0x5632dc862380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.262+0000 7f586bba4640 -1 rbd::mirror::MirrorStatusUpdater 0x5632dc8c8000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.262+0000 7f586bba4640 -1 rbd::mirror::MirrorStatusUpdater 0x5632dc8c8120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.262+0000 7f58723b1640 -1 rbd::mirror::NamespaceReplayer: 0x5632dc8c6000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.260 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.262+0000 7f58723b1640 -1 rbd::mirror::NamespaceReplayer: 0x5632dc8c61a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.263 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.262+0000 7f5872bb2640 -1 rbd::mirror::LeaderWatcher: 0x5632dc7b6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.265 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.266+0000 7f5872bb2640 -1 rbd::mirror::InstanceWatcher: 0x5632dc862000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6102 object: (2) No such file or directory 2024-09-11T09:15:12.266 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.266+0000 7f586c3a5640 -1 rbd::mirror::InstanceWatcher: 0x5632dc862000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.266+0000 7f586bba4640 -1 rbd::mirror::MirrorStatusUpdater 0x5632dd2350e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.267 INFO:tasks.rbd_mirror.cluster1.client.mirror.2.smithi077.stderr:2024-09-11T09:15:12.266+0000 7f5872bb2640 -1 rbd::mirror::NamespaceReplayer: 0x5632dc5ce340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.292 INFO:tasks.rbd_mirror.cluster1.client.mirror.2:Stopped 2024-09-11T09:15:12.292 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:12.302 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.1:waiting for process to exit 2024-09-11T09:15:12.302 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:12.336 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.338+0000 7f22e7373640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.1 (PID: 63160) UID: 1000 2024-09-11T09:15:12.338 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.338+0000 7f22e6371640 -1 rbd::mirror::LeaderWatcher: 0x558751d05200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.338+0000 7f22e6b72640 -1 rbd::mirror::InstanceWatcher: 0x558751ac8380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5985 object: (2) No such file or directory 2024-09-11T09:15:12.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.338+0000 7f22e6b72640 -1 rbd::mirror::InstanceWatcher: 0x558751ac8a80 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5985 object: (2) No such file or directory 2024-09-11T09:15:12.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.338+0000 7f22e6371640 -1 rbd::mirror::InstanceWatcher: 0x558751fc2000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5988 object: (2) No such file or directory 2024-09-11T09:15:12.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.338+0000 7f22dfb64640 -1 rbd::mirror::InstanceWatcher: 0x558751ac8380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.339 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.338+0000 7f22dfb64640 -1 rbd::mirror::InstanceWatcher: 0x558751ac8a80 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.340 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22d2b4a640 -1 rbd::mirror::InstanceWatcher: 0x558751fc2000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22d9b58640 -1 rbd::mirror::MirrorStatusUpdater 0x558751b32360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22d9b58640 -1 rbd::mirror::MirrorStatusUpdater 0x558751b325a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e6b72640 -1 rbd::mirror::NamespaceReplayer: 0x558751b2e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e6b72640 -1 rbd::mirror::NamespaceReplayer: 0x558751b2e1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e0365640 -1 rbd::mirror::MirrorStatusUpdater 0x558751b32000 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.341 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22cc33d640 -1 rbd::mirror::MirrorStatusUpdater 0x558751d347e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e0365640 -1 rbd::mirror::MirrorStatusUpdater 0x558751b32120 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e6371640 -1 rbd::mirror::NamespaceReplayer: 0x558751b2e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e6371640 -1 rbd::mirror::NamespaceReplayer: 0x558751fbc000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e6371640 -1 rbd::mirror::NamespaceReplayer: 0x558751b2e1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22d2349640 -1 rbd::mirror::MirrorStatusUpdater 0x558751d010e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.342 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.342+0000 7f22e6371640 -1 rbd::mirror::NamespaceReplayer: 0x558751fbc000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.346 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.346+0000 7f22e6371640 -1 rbd::mirror::LeaderWatcher: 0x5587528c6c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.346+0000 7f22e6371640 -1 rbd::mirror::InstanceWatcher: 0x558751ac8000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.5985 object: (2) No such file or directory 2024-09-11T09:15:12.347 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.346+0000 7f22dfb64640 -1 rbd::mirror::InstanceWatcher: 0x558751ac8000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.350+0000 7f22d9357640 -1 rbd::mirror::MirrorStatusUpdater 0x5587527126c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.348 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.350+0000 7f22e6371640 -1 rbd::mirror::NamespaceReplayer: 0x558751a4a1a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.350+0000 7f22e0365640 -1 rbd::mirror::MirrorStatusUpdater 0x558752694fc0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.349 INFO:tasks.rbd_mirror.cluster1.client.mirror.1.smithi077.stderr:2024-09-11T09:15:12.350+0000 7f22e6b72640 -1 rbd::mirror::NamespaceReplayer: 0x558751a4a1a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.373 INFO:tasks.rbd_mirror.cluster1.client.mirror.1:Stopped 2024-09-11T09:15:12.373 DEBUG:teuthology.run_tasks:Unwinding manager rbd-mirror 2024-09-11T09:15:12.384 DEBUG:tasks.rbd_mirror.cluster1.client.mirror.0:waiting for process to exit 2024-09-11T09:15:12.384 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:12.416 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.418+0000 7f8088fba640 -1 received signal: Terminated from /usr/bin/python3 /usr/bin/daemon-helper term rbd-mirror --foreground --cluster cluster1 --id mirror.0 (PID: 66495) UID: 1000 2024-09-11T09:15:12.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.418+0000 7f8087fb8640 -1 rbd::mirror::LeaderWatcher: 0x5593fd1cd200 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.418 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.418+0000 7f80887b9640 -1 rbd::mirror::InstanceWatcher: 0x5593fcf78700 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6134 object: (2) No such file or directory 2024-09-11T09:15:12.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.418+0000 7f80887b9640 -1 rbd::mirror::InstanceWatcher: 0x5593fcf78380 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6134 object: (2) No such file or directory 2024-09-11T09:15:12.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.418+0000 7f80887b9640 -1 rbd::mirror::InstanceWatcher: 0x5593fd472000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6137 object: (2) No such file or directory 2024-09-11T09:15:12.419 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.418+0000 7f80817ab640 -1 rbd::mirror::InstanceWatcher: 0x5593fcf78700 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.418+0000 7f80817ab640 -1 rbd::mirror::InstanceWatcher: 0x5593fcf78380 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f8074791640 -1 rbd::mirror::InstanceWatcher: 0x5593fd472000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.420 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f807b79f640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fcfae7e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f807b79f640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fcfae5a0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f8087fb8640 -1 rbd::mirror::NamespaceReplayer: 0x5593fcff21a0 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f806df84640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fcfaed80 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f8087fb8640 -1 rbd::mirror::NamespaceReplayer: 0x5593fcff2000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f8087fb8640 -1 rbd::mirror::NamespaceReplayer: 0x5593fd46e000 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f8081fac640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fcfae360 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.421 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f80817ab640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fcfae240 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f8073f90640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fd1b1560 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f80887b9640 -1 rbd::mirror::NamespaceReplayer: 0x5593fcff21a0 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f80887b9640 -1 rbd::mirror::NamespaceReplayer: 0x5593fcff2000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.422 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.422+0000 7f80887b9640 -1 rbd::mirror::NamespaceReplayer: 0x5593fd46e000 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.425 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.426+0000 7f8087fb8640 -1 rbd::mirror::LeaderWatcher: 0x5593fced8c00 handle_unregister_watch: error unregistering leader watcher for rbd_mirror_leader object: (2) No such file or directory 2024-09-11T09:15:12.426 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.426+0000 7f8087fb8640 -1 rbd::mirror::InstanceWatcher: 0x5593fcf78000 handle_unregister_watch: error unregistering instance watcher for rbd_mirror_instance.6134 object: (2) No such file or directory 2024-09-11T09:15:12.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.426+0000 7f8081fac640 -1 rbd::mirror::InstanceWatcher: 0x5593fcf78000 handle_unregister_instance: error unregistering instance: (2) No such file or directory 2024-09-11T09:15:12.427 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.426+0000 7f807b79f640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fd9ca6c0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.426+0000 7f8087fb8640 -1 rbd::mirror::NamespaceReplayer: 0x5593fcce8340 handle_shut_down_remote_status_updater: error shutting remote mirror status updater down: (2) No such file or directory 2024-09-11T09:15:12.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.430+0000 7f80817ab640 -1 rbd::mirror::MirrorStatusUpdater 0x5593fd94d0e0 handle_shut_down_mirror_status_watcher: failed to shut down mirror status watcher: (2) No such file or directory 2024-09-11T09:15:12.428 INFO:tasks.rbd_mirror.cluster1.client.mirror.0.smithi077.stderr:2024-09-11T09:15:12.430+0000 7f80887b9640 -1 rbd::mirror::NamespaceReplayer: 0x5593fcce8340 handle_shut_down_local_status_updater: error shutting local mirror status updater down: (2) No such file or directory 2024-09-11T09:15:18.486 INFO:tasks.rbd_mirror.cluster1.client.mirror.0:Stopped 2024-09-11T09:15:18.486 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T09:15:18.496 INFO:tasks.ceph.ceph_manager.cluster2:waiting for clean 2024-09-11T09:15:18.496 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T09:15:18.805 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T09:15:18.806 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-09-11T09:15:18.823 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1532,"stamp":"2024-09-11T09:15:18.785227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2680064,"kb_used_data":16960,"kb_used_omap":302,"kb_used_meta":2662737,"kb_avail":280435456,"statfs":{"total":289910292480,"available":287165906944,"internally_reserved":0,"allocated":17367040,"data_stored":9098434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":309671,"internal_metadata":2726643289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2071,"num_read_kb":-1960,"num_write":-394,"num_write_kb":-304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.833607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414779+0000","last_change":"2024-09-11T08:39:27.014675+0000","last_active":"2024-09-11T09:15:11.414779+0000","last_peered":"2024-09-11T09:15:11.414779+0000","last_clean":"2024-09-11T09:15:11.414779+0000","last_became_active":"2024-09-11T08:39:24.993902+0000","last_became_peered":"2024-09-11T08:39:24.993902+0000","last_unstale":"2024-09-11T09:15:11.414779+0000","last_undegraded":"2024-09-11T09:15:11.414779+0000","last_fullsized":"2024-09-11T09:15:11.414779+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:22:21.610509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.415011+0000","last_change":"2024-09-11T08:39:27.015272+0000","last_active":"2024-09-11T09:15:11.415011+0000","last_peered":"2024-09-11T09:15:11.415011+0000","last_clean":"2024-09-11T09:15:11.415011+0000","last_became_active":"2024-09-11T08:39:24.995216+0000","last_became_peered":"2024-09-11T08:39:24.995216+0000","last_unstale":"2024-09-11T09:15:11.415011+0000","last_undegraded":"2024-09-11T09:15:11.415011+0000","last_fullsized":"2024-09-11T09:15:11.415011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:45:49.872594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414856+0000","last_change":"2024-09-11T08:39:27.014087+0000","last_active":"2024-09-11T09:15:11.414856+0000","last_peered":"2024-09-11T09:15:11.414856+0000","last_clean":"2024-09-11T09:15:11.414856+0000","last_became_active":"2024-09-11T08:39:24.994002+0000","last_became_peered":"2024-09-11T08:39:24.994002+0000","last_unstale":"2024-09-11T09:15:11.414856+0000","last_undegraded":"2024-09-11T09:15:11.414856+0000","last_fullsized":"2024-09-11T09:15:11.414856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:54:18.729378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070708900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.417014+0000","last_change":"2024-09-11T08:39:27.014269+0000","last_active":"2024-09-11T09:15:11.417014+0000","last_peered":"2024-09-11T09:15:11.417014+0000","last_clean":"2024-09-11T09:15:11.417014+0000","last_became_active":"2024-09-11T08:39:24.995202+0000","last_became_peered":"2024-09-11T08:39:24.995202+0000","last_unstale":"2024-09-11T09:15:11.417014+0000","last_undegraded":"2024-09-11T09:15:11.417014+0000","last_fullsized":"2024-09-11T09:15:11.417014+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:38.241377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010507380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.395116+0000","last_change":"2024-09-11T08:39:27.008400+0000","last_active":"2024-09-11T09:15:11.395116+0000","last_peered":"2024-09-11T09:15:11.395116+0000","last_clean":"2024-09-11T09:15:11.395116+0000","last_became_active":"2024-09-11T08:39:24.990422+0000","last_became_peered":"2024-09-11T08:39:24.990422+0000","last_unstale":"2024-09-11T09:15:11.395116+0000","last_undegraded":"2024-09-11T09:15:11.395116+0000","last_fullsized":"2024-09-11T09:15:11.395116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:59:33.107916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000934803,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.960990+0000","last_change":"2024-09-11T08:39:27.657840+0000","last_active":"2024-09-11T09:15:10.960990+0000","last_peered":"2024-09-11T09:15:10.960990+0000","last_clean":"2024-09-11T09:15:10.960990+0000","last_became_active":"2024-09-11T08:39:24.994237+0000","last_became_peered":"2024-09-11T08:39:24.994237+0000","last_unstale":"2024-09-11T09:15:10.960990+0000","last_undegraded":"2024-09-11T09:15:10.960990+0000","last_fullsized":"2024-09-11T09:15:10.960990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:53.237878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063005199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.962738+0000","last_change":"2024-09-11T08:39:27.657633+0000","last_active":"2024-09-11T09:15:10.962738+0000","last_peered":"2024-09-11T09:15:10.962738+0000","last_clean":"2024-09-11T09:15:10.962738+0000","last_became_active":"2024-09-11T08:39:24.990604+0000","last_became_peered":"2024-09-11T08:39:24.990604+0000","last_unstale":"2024-09-11T09:15:10.962738+0000","last_undegraded":"2024-09-11T09:15:10.962738+0000","last_fullsized":"2024-09-11T09:15:10.962738+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:38.112533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000543883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416251+0000","last_change":"2024-09-11T08:39:27.015047+0000","last_active":"2024-09-11T09:15:11.416251+0000","last_peered":"2024-09-11T09:15:11.416251+0000","last_clean":"2024-09-11T09:15:11.416251+0000","last_became_active":"2024-09-11T08:39:24.987701+0000","last_became_peered":"2024-09-11T08:39:24.987701+0000","last_unstale":"2024-09-11T09:15:11.416251+0000","last_undegraded":"2024-09-11T09:15:11.416251+0000","last_fullsized":"2024-09-11T09:15:11.416251+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:55:50.605878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016047609999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1042,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416498+0000","last_change":"2024-09-11T08:39:22.547555+0000","last_active":"2024-09-11T09:15:11.416498+0000","last_peered":"2024-09-11T09:15:11.416498+0000","last_clean":"2024-09-11T09:15:11.416498+0000","last_became_active":"2024-09-11T08:39:22.544974+0000","last_became_peered":"2024-09-11T08:39:22.544974+0000","last_unstale":"2024-09-11T09:15:11.416498+0000","last_undegraded":"2024-09-11T09:15:11.416498+0000","last_fullsized":"2024-09-11T09:15:11.416498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_clean_scrub_stamp":"2024-09-11T08:39:20.961806+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:01.190641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838952,"kb_used_data":5264,"kb_used_omap":101,"kb_used_meta":833562,"kb_avail":93532888,"statfs":{"total":96636764160,"available":95777677312,"internally_reserved":0,"allocated":5390336,"data_stored":2639220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103937,"internal_metadata":853567999},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717356,"kb_used_data":5848,"kb_used_omap":95,"kb_used_meta":711392,"kb_avail":93654484,"statfs":{"total":96636764160,"available":95902191616,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97785,"internal_metadata":728465927},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738803,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1123756,"kb_used_data":5848,"kb_used_omap":105,"kb_used_meta":1117782,"kb_avail":93248084,"statfs":{"total":96636764160,"available":95486038016,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107949,"internal_metadata":1144609363},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:15:18.823 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T09:15:19.094 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T09:15:19.095 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-09-11T09:15:19.111 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1532,"stamp":"2024-09-11T09:15:18.785227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2680064,"kb_used_data":16960,"kb_used_omap":302,"kb_used_meta":2662737,"kb_avail":280435456,"statfs":{"total":289910292480,"available":287165906944,"internally_reserved":0,"allocated":17367040,"data_stored":9098434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":309671,"internal_metadata":2726643289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2071,"num_read_kb":-1960,"num_write":-394,"num_write_kb":-304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.833607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414779+0000","last_change":"2024-09-11T08:39:27.014675+0000","last_active":"2024-09-11T09:15:11.414779+0000","last_peered":"2024-09-11T09:15:11.414779+0000","last_clean":"2024-09-11T09:15:11.414779+0000","last_became_active":"2024-09-11T08:39:24.993902+0000","last_became_peered":"2024-09-11T08:39:24.993902+0000","last_unstale":"2024-09-11T09:15:11.414779+0000","last_undegraded":"2024-09-11T09:15:11.414779+0000","last_fullsized":"2024-09-11T09:15:11.414779+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:22:21.610509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.415011+0000","last_change":"2024-09-11T08:39:27.015272+0000","last_active":"2024-09-11T09:15:11.415011+0000","last_peered":"2024-09-11T09:15:11.415011+0000","last_clean":"2024-09-11T09:15:11.415011+0000","last_became_active":"2024-09-11T08:39:24.995216+0000","last_became_peered":"2024-09-11T08:39:24.995216+0000","last_unstale":"2024-09-11T09:15:11.415011+0000","last_undegraded":"2024-09-11T09:15:11.415011+0000","last_fullsized":"2024-09-11T09:15:11.415011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:45:49.872594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414856+0000","last_change":"2024-09-11T08:39:27.014087+0000","last_active":"2024-09-11T09:15:11.414856+0000","last_peered":"2024-09-11T09:15:11.414856+0000","last_clean":"2024-09-11T09:15:11.414856+0000","last_became_active":"2024-09-11T08:39:24.994002+0000","last_became_peered":"2024-09-11T08:39:24.994002+0000","last_unstale":"2024-09-11T09:15:11.414856+0000","last_undegraded":"2024-09-11T09:15:11.414856+0000","last_fullsized":"2024-09-11T09:15:11.414856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:54:18.729378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070708900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.417014+0000","last_change":"2024-09-11T08:39:27.014269+0000","last_active":"2024-09-11T09:15:11.417014+0000","last_peered":"2024-09-11T09:15:11.417014+0000","last_clean":"2024-09-11T09:15:11.417014+0000","last_became_active":"2024-09-11T08:39:24.995202+0000","last_became_peered":"2024-09-11T08:39:24.995202+0000","last_unstale":"2024-09-11T09:15:11.417014+0000","last_undegraded":"2024-09-11T09:15:11.417014+0000","last_fullsized":"2024-09-11T09:15:11.417014+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:38.241377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010507380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.395116+0000","last_change":"2024-09-11T08:39:27.008400+0000","last_active":"2024-09-11T09:15:11.395116+0000","last_peered":"2024-09-11T09:15:11.395116+0000","last_clean":"2024-09-11T09:15:11.395116+0000","last_became_active":"2024-09-11T08:39:24.990422+0000","last_became_peered":"2024-09-11T08:39:24.990422+0000","last_unstale":"2024-09-11T09:15:11.395116+0000","last_undegraded":"2024-09-11T09:15:11.395116+0000","last_fullsized":"2024-09-11T09:15:11.395116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:59:33.107916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000934803,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.960990+0000","last_change":"2024-09-11T08:39:27.657840+0000","last_active":"2024-09-11T09:15:10.960990+0000","last_peered":"2024-09-11T09:15:10.960990+0000","last_clean":"2024-09-11T09:15:10.960990+0000","last_became_active":"2024-09-11T08:39:24.994237+0000","last_became_peered":"2024-09-11T08:39:24.994237+0000","last_unstale":"2024-09-11T09:15:10.960990+0000","last_undegraded":"2024-09-11T09:15:10.960990+0000","last_fullsized":"2024-09-11T09:15:10.960990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:53.237878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063005199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.962738+0000","last_change":"2024-09-11T08:39:27.657633+0000","last_active":"2024-09-11T09:15:10.962738+0000","last_peered":"2024-09-11T09:15:10.962738+0000","last_clean":"2024-09-11T09:15:10.962738+0000","last_became_active":"2024-09-11T08:39:24.990604+0000","last_became_peered":"2024-09-11T08:39:24.990604+0000","last_unstale":"2024-09-11T09:15:10.962738+0000","last_undegraded":"2024-09-11T09:15:10.962738+0000","last_fullsized":"2024-09-11T09:15:10.962738+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:38.112533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000543883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416251+0000","last_change":"2024-09-11T08:39:27.015047+0000","last_active":"2024-09-11T09:15:11.416251+0000","last_peered":"2024-09-11T09:15:11.416251+0000","last_clean":"2024-09-11T09:15:11.416251+0000","last_became_active":"2024-09-11T08:39:24.987701+0000","last_became_peered":"2024-09-11T08:39:24.987701+0000","last_unstale":"2024-09-11T09:15:11.416251+0000","last_undegraded":"2024-09-11T09:15:11.416251+0000","last_fullsized":"2024-09-11T09:15:11.416251+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:55:50.605878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016047609999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1042,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416498+0000","last_change":"2024-09-11T08:39:22.547555+0000","last_active":"2024-09-11T09:15:11.416498+0000","last_peered":"2024-09-11T09:15:11.416498+0000","last_clean":"2024-09-11T09:15:11.416498+0000","last_became_active":"2024-09-11T08:39:22.544974+0000","last_became_peered":"2024-09-11T08:39:22.544974+0000","last_unstale":"2024-09-11T09:15:11.416498+0000","last_undegraded":"2024-09-11T09:15:11.416498+0000","last_fullsized":"2024-09-11T09:15:11.416498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_clean_scrub_stamp":"2024-09-11T08:39:20.961806+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:01.190641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838952,"kb_used_data":5264,"kb_used_omap":101,"kb_used_meta":833562,"kb_avail":93532888,"statfs":{"total":96636764160,"available":95777677312,"internally_reserved":0,"allocated":5390336,"data_stored":2639220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103937,"internal_metadata":853567999},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717356,"kb_used_data":5848,"kb_used_omap":95,"kb_used_meta":711392,"kb_avail":93654484,"statfs":{"total":96636764160,"available":95902191616,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97785,"internal_metadata":728465927},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738803,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1123756,"kb_used_data":5848,"kb_used_omap":105,"kb_used_meta":1117782,"kb_avail":93248084,"statfs":{"total":96636764160,"available":95486038016,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107949,"internal_metadata":1144609363},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:15:19.112 INFO:tasks.ceph.ceph_manager.cluster2:clean! 2024-09-11T09:15:19.112 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T09:15:19.407 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T09:15:19.408 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-09-11T09:15:19.426 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1532,"stamp":"2024-09-11T09:15:18.785227+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2680064,"kb_used_data":16960,"kb_used_omap":302,"kb_used_meta":2662737,"kb_avail":280435456,"statfs":{"total":289910292480,"available":287165906944,"internally_reserved":0,"allocated":17367040,"data_stored":9098434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":309671,"internal_metadata":2726643289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":5,"apply_latency_ms":5,"commit_latency_ns":5000000,"apply_latency_ns":5000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2071,"num_read_kb":-1960,"num_write":-394,"num_write_kb":-304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"9.833607"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414779+0000","last_change":"2024-09-11T08:39:27.014675+0000","last_active":"2024-09-11T09:15:11.414779+0000","last_peered":"2024-09-11T09:15:11.414779+0000","last_clean":"2024-09-11T09:15:11.414779+0000","last_became_active":"2024-09-11T08:39:24.993902+0000","last_became_peered":"2024-09-11T08:39:24.993902+0000","last_unstale":"2024-09-11T09:15:11.414779+0000","last_undegraded":"2024-09-11T09:15:11.414779+0000","last_fullsized":"2024-09-11T09:15:11.414779+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:22:21.610509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.415011+0000","last_change":"2024-09-11T08:39:27.015272+0000","last_active":"2024-09-11T09:15:11.415011+0000","last_peered":"2024-09-11T09:15:11.415011+0000","last_clean":"2024-09-11T09:15:11.415011+0000","last_became_active":"2024-09-11T08:39:24.995216+0000","last_became_peered":"2024-09-11T08:39:24.995216+0000","last_unstale":"2024-09-11T09:15:11.415011+0000","last_undegraded":"2024-09-11T09:15:11.415011+0000","last_fullsized":"2024-09-11T09:15:11.415011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:45:49.872594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414856+0000","last_change":"2024-09-11T08:39:27.014087+0000","last_active":"2024-09-11T09:15:11.414856+0000","last_peered":"2024-09-11T09:15:11.414856+0000","last_clean":"2024-09-11T09:15:11.414856+0000","last_became_active":"2024-09-11T08:39:24.994002+0000","last_became_peered":"2024-09-11T08:39:24.994002+0000","last_unstale":"2024-09-11T09:15:11.414856+0000","last_undegraded":"2024-09-11T09:15:11.414856+0000","last_fullsized":"2024-09-11T09:15:11.414856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:54:18.729378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070708900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.417014+0000","last_change":"2024-09-11T08:39:27.014269+0000","last_active":"2024-09-11T09:15:11.417014+0000","last_peered":"2024-09-11T09:15:11.417014+0000","last_clean":"2024-09-11T09:15:11.417014+0000","last_became_active":"2024-09-11T08:39:24.995202+0000","last_became_peered":"2024-09-11T08:39:24.995202+0000","last_unstale":"2024-09-11T09:15:11.417014+0000","last_undegraded":"2024-09-11T09:15:11.417014+0000","last_fullsized":"2024-09-11T09:15:11.417014+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:38.241377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010507380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.395116+0000","last_change":"2024-09-11T08:39:27.008400+0000","last_active":"2024-09-11T09:15:11.395116+0000","last_peered":"2024-09-11T09:15:11.395116+0000","last_clean":"2024-09-11T09:15:11.395116+0000","last_became_active":"2024-09-11T08:39:24.990422+0000","last_became_peered":"2024-09-11T08:39:24.990422+0000","last_unstale":"2024-09-11T09:15:11.395116+0000","last_undegraded":"2024-09-11T09:15:11.395116+0000","last_fullsized":"2024-09-11T09:15:11.395116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:59:33.107916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000934803,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.960990+0000","last_change":"2024-09-11T08:39:27.657840+0000","last_active":"2024-09-11T09:15:10.960990+0000","last_peered":"2024-09-11T09:15:10.960990+0000","last_clean":"2024-09-11T09:15:10.960990+0000","last_became_active":"2024-09-11T08:39:24.994237+0000","last_became_peered":"2024-09-11T08:39:24.994237+0000","last_unstale":"2024-09-11T09:15:10.960990+0000","last_undegraded":"2024-09-11T09:15:10.960990+0000","last_fullsized":"2024-09-11T09:15:10.960990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:53.237878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063005199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.962738+0000","last_change":"2024-09-11T08:39:27.657633+0000","last_active":"2024-09-11T09:15:10.962738+0000","last_peered":"2024-09-11T09:15:10.962738+0000","last_clean":"2024-09-11T09:15:10.962738+0000","last_became_active":"2024-09-11T08:39:24.990604+0000","last_became_peered":"2024-09-11T08:39:24.990604+0000","last_unstale":"2024-09-11T09:15:10.962738+0000","last_undegraded":"2024-09-11T09:15:10.962738+0000","last_fullsized":"2024-09-11T09:15:10.962738+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:38.112533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000543883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416251+0000","last_change":"2024-09-11T08:39:27.015047+0000","last_active":"2024-09-11T09:15:11.416251+0000","last_peered":"2024-09-11T09:15:11.416251+0000","last_clean":"2024-09-11T09:15:11.416251+0000","last_became_active":"2024-09-11T08:39:24.987701+0000","last_became_peered":"2024-09-11T08:39:24.987701+0000","last_unstale":"2024-09-11T09:15:11.416251+0000","last_undegraded":"2024-09-11T09:15:11.416251+0000","last_fullsized":"2024-09-11T09:15:11.416251+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:55:50.605878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016047609999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1042,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416498+0000","last_change":"2024-09-11T08:39:22.547555+0000","last_active":"2024-09-11T09:15:11.416498+0000","last_peered":"2024-09-11T09:15:11.416498+0000","last_clean":"2024-09-11T09:15:11.416498+0000","last_became_active":"2024-09-11T08:39:22.544974+0000","last_became_peered":"2024-09-11T08:39:22.544974+0000","last_unstale":"2024-09-11T09:15:11.416498+0000","last_undegraded":"2024-09-11T09:15:11.416498+0000","last_fullsized":"2024-09-11T09:15:11.416498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_clean_scrub_stamp":"2024-09-11T08:39:20.961806+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:01.190641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738803,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838952,"kb_used_data":5264,"kb_used_omap":101,"kb_used_meta":833562,"kb_avail":93532888,"statfs":{"total":96636764160,"available":95777677312,"internally_reserved":0,"allocated":5390336,"data_stored":2639220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103937,"internal_metadata":853567999},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":3,"apply_latency_ms":3,"commit_latency_ns":3000000,"apply_latency_ns":3000000},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717356,"kb_used_data":5848,"kb_used_omap":95,"kb_used_meta":711392,"kb_avail":93654484,"statfs":{"total":96636764160,"available":95902191616,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97785,"internal_metadata":728465927},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738803,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1123756,"kb_used_data":5848,"kb_used_omap":105,"kb_used_meta":1117782,"kb_avail":93248084,"statfs":{"total":96636764160,"available":95486038016,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107949,"internal_metadata":1144609363},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:15:19.427 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd dump --format=json 2024-09-11T09:15:19.706 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T09:15:19.706 INFO:teuthology.orchestra.run.smithi077.stdout:{"epoch":455,"fsid":"600c207a-6d49-48ae-9606-243c4786b265","created":"2024-09-11T08:39:08.452088+0000","modified":"2024-09-11T09:15:10.951168+0000","last_up_change":"2024-09-11T08:39:19.960494+0000","last_in_change":"2024-09-11T08:39:09.383976+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:39:20.054791+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T08:39:23.746149+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"5f59c3b8-aae6-4ec2-b2b7-f2f6fcde5ef7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6816","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6817","nonce":167757817}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6818","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6819","nonce":167757817}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6822","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6823","nonce":167757817}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6820","nonce":167757817},{"type":"v1","addr":"172.21.15.77:6821","nonce":167757817}]},"public_addr":"172.21.15.77:6817/167757817","cluster_addr":"172.21.15.77:6819/167757817","heartbeat_back_addr":"172.21.15.77:6823/167757817","heartbeat_front_addr":"172.21.15.77:6821/167757817","state":["exists","up"]},{"osd":1,"uuid":"f60312d0-3b69-4990-9cb6-df08bf6608b7","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6800","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6801","nonce":3684380181}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6802","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6803","nonce":3684380181}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6806","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6807","nonce":3684380181}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6804","nonce":3684380181},{"type":"v1","addr":"172.21.15.77:6805","nonce":3684380181}]},"public_addr":"172.21.15.77:6801/3684380181","cluster_addr":"172.21.15.77:6803/3684380181","heartbeat_back_addr":"172.21.15.77:6807/3684380181","heartbeat_front_addr":"172.21.15.77:6805/3684380181","state":["exists","up"]},{"osd":2,"uuid":"9675147d-7ef3-4ef9-bb45-278bb1c44609","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6808","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6809","nonce":1617494006}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6810","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6811","nonce":1617494006}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6814","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6815","nonce":1617494006}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.77:6812","nonce":1617494006},{"type":"v1","addr":"172.21.15.77:6813","nonce":1617494006}]},"public_addr":"172.21.15.77:6809/1617494006","cluster_addr":"172.21.15.77:6811/1617494006","heartbeat_back_addr":"172.21.15.77:6815/1617494006","heartbeat_front_addr":"172.21.15.77:6813/1617494006","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.433046+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.136998+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:39:16.660776+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.77:0/3003222219":"2024-09-11T10:14:28.147501+0000","172.21.15.77:0/1375065491":"2024-09-11T10:13:47.979971+0000","172.21.15.77:0/4108385269":"2024-09-11T10:13:11.555523+0000","172.21.15.77:0/1987112296":"2024-09-11T10:11:57.782631+0000","172.21.15.77:0/4062319768":"2024-09-11T10:11:13.920935+0000","172.21.15.77:0/3720662838":"2024-09-11T10:12:33.744737+0000","172.21.15.77:0/280448440":"2024-09-11T10:10:00.362431+0000","172.21.15.77:0/1721960457":"2024-09-11T10:01:54.111599+0000","172.21.15.77:0/663690627":"2024-09-11T10:02:01.435280+0000","172.21.15.77:0/2778637358":"2024-09-11T10:06:19.379448+0000","172.21.15.77:0/2085493023":"2024-09-11T09:43:43.578355+0000","172.21.15.77:0/2422209865":"2024-09-11T10:01:45.409577+0000","172.21.15.77:0/1581819197":"2024-09-11T10:04:04.867435+0000","172.21.15.77:0/1734274620":"2024-09-11T10:01:10.202642+0000","172.21.15.77:0/2947926287":"2024-09-11T10:01:02.800810+0000","172.21.15.77:0/163129157":"2024-09-11T10:00:19.118288+0000","172.21.15.77:0/3344358339":"2024-09-11T10:02:08.067662+0000","172.21.15.77:0/446979137":"2024-09-11T10:01:31.264833+0000","172.21.15.77:0/1915815211":"2024-09-11T09:48:55.765980+0000","172.21.15.77:0/3314160009":"2024-09-11T10:03:17.599481+0000","172.21.15.77:0/2486521724":"2024-09-11T10:05:49.368270+0000","172.21.15.77:0/2923139487":"2024-09-11T10:03:56.082481+0000","172.21.15.77:0/657512178":"2024-09-11T10:05:19.322854+0000","172.21.15.77:0/163340294":"2024-09-11T09:41:59.138631+0000","172.21.15.77:0/2966302410":"2024-09-11T09:44:33.645569+0000","172.21.15.77:0/2139764570":"2024-09-11T10:01:17.440102+0000","172.21.15.77:0/1971666589":"2024-09-11T09:59:56.109289+0000","172.21.15.77:0/2804733703":"2024-09-11T10:03:09.473934+0000","172.21.15.77:0/2987634046":"2024-09-11T10:02:42.194290+0000","172.21.15.77:0/2221435023":"2024-09-11T10:04:33.234564+0000","172.21.15.77:0/567214197":"2024-09-11T10:09:17.609276+0000","172.21.15.77:0/1022640064":"2024-09-11T10:01:38.349865+0000","172.21.15.77:0/1549473111":"2024-09-11T10:00:03.351264+0000","172.21.15.77:0/3665731907":"2024-09-11T09:45:29.901252+0000","172.21.15.77:0/3134835101":"2024-09-11T10:00:27.723687+0000","172.21.15.77:0/3446182724":"2024-09-11T09:46:21.114317+0000","172.21.15.77:0/3288375393":"2024-09-11T10:02:34.808447+0000","172.21.15.77:0/646627831":"2024-09-11T09:49:45.625304+0000","172.21.15.77:0/3899343609":"2024-09-11T09:47:10.780358+0000","172.21.15.77:0/2144008193":"2024-09-11T09:48:00.813435+0000","172.21.15.77:0/1760255791":"2024-09-11T10:04:19.245577+0000","172.21.15.77:0/3317247399":"2024-09-11T09:42:51.571202+0000","172.21.15.77:0/1984524241":"2024-09-11T10:10:37.132209+0000","172.21.15.77:0/765029679":"2024-09-11T10:02:16.778811+0000","172.21.15.77:0/1063700397":"2024-09-11T10:04:26.224625+0000","172.21.15.77:0/1128434868":"2024-09-11T10:00:10.406089+0000","172.21.15.77:0/2395874350":"2024-09-11T10:02:49.182590+0000","172.21.15.77:0/1677872915":"2024-09-11T10:03:40.020908+0000","172.21.15.77:0/1593455612":"2024-09-11T10:02:56.285778+0000","172.21.15.77:0/265508312":"2024-09-11T10:03:24.378859+0000","172.21.15.77:0/3337108388":"2024-09-11T10:03:31.404236+0000","172.21.15.77:0/3624186319":"2024-09-11T10:03:49.165641+0000","172.21.15.77:0/1364627049":"2024-09-11T10:04:12.212966+0000","172.21.15.77:0/1929047005":"2024-09-11T10:04:40.571749+0000","172.21.15.77:0/2327367340":"2024-09-11T10:15:06.688083+0000","172.21.15.77:0/1622557076":"2024-09-11T10:04:47.351271+0000","172.21.15.77:0/2928341574":"2024-09-11T10:06:40.543280+0000","172.21.15.77:0/4020574979":"2024-09-11T10:05:10.368264+0000","172.21.15.77:0/1706521391":"2024-09-11T10:04:54.387296+0000","172.21.15.77:0/1296872757":"2024-09-11T10:05:01.323765+0000","172.21.15.77:0/2222202238":"2024-09-11T10:05:26.391214+0000","172.21.15.77:0/3992038948":"2024-09-11T10:05:34.921417+0000","172.21.15.77:0/1325652316":"2024-09-11T10:05:42.166349+0000","172.21.15.77:0/1925685940":"2024-09-11T10:06:03.465330+0000","172.21.15.77:0/4232444814":"2024-09-11T10:05:56.238730+0000","172.21.15.77:0/2952342437":"2024-09-11T10:02:25.878908+0000","172.21.15.77:0/1560924939":"2024-09-11T10:06:12.108323+0000","172.21.15.77:0/1508971309":"2024-09-11T10:06:26.379535+0000","172.21.15.77:0/3825242490":"2024-09-11T10:01:24.228595+0000","172.21.15.77:0/2141011070":"2024-09-11T10:06:33.407879+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:15:20.724 INFO:tasks.ceph:Scrubbing cluster2.osd.0 2024-09-11T09:15:20.724 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:15:20.882 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-09-11T09:15:20.882 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:15:20.882 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-09-11T09:15:20.898 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 0 2024-09-11T09:15:21.190 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T09:15:21.209 INFO:tasks.ceph:Scrubbing cluster2.osd.1 2024-09-11T09:15:21.209 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:15:21.346 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-09-11T09:15:21.346 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:15:21.346 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-09-11T09:15:21.361 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 1 2024-09-11T09:15:21.645 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T09:15:21.663 INFO:tasks.ceph:Scrubbing cluster2.osd.2 2024-09-11T09:15:21.663 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:15:21.802 INFO:teuthology.orchestra.run.smithi077.stdout:{ 2024-09-11T09:15:21.802 INFO:teuthology.orchestra.run.smithi077.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:15:21.802 INFO:teuthology.orchestra.run.smithi077.stdout:} 2024-09-11T09:15:21.817 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 osd deep-scrub 2 2024-09-11T09:15:22.099 INFO:teuthology.orchestra.run.smithi077.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T09:15:22.117 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T09:15:22.395 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T09:15:22.395 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-09-11T09:15:22.412 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1533,"stamp":"2024-09-11T09:15:20.785768+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2680064,"kb_used_data":16960,"kb_used_omap":302,"kb_used_meta":2662737,"kb_avail":280435456,"statfs":{"total":289910292480,"available":287165906944,"internally_reserved":0,"allocated":17367040,"data_stored":9098434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":309671,"internal_metadata":2726643289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-19,"num_objects":-8,"num_object_clones":0,"num_object_copies":-16,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-8,"num_whiteouts":0,"num_read":-2075,"num_read_kb":-1964,"num_write":-394,"num_write_kb":-304,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":-1,"num_bytes_recovered":0,"num_keys_recovered":-3,"num_objects_omap":-2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"10.002961"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414779+0000","last_change":"2024-09-11T08:39:27.014675+0000","last_active":"2024-09-11T09:15:11.414779+0000","last_peered":"2024-09-11T09:15:11.414779+0000","last_clean":"2024-09-11T09:15:11.414779+0000","last_became_active":"2024-09-11T08:39:24.993902+0000","last_became_peered":"2024-09-11T08:39:24.993902+0000","last_unstale":"2024-09-11T09:15:11.414779+0000","last_undegraded":"2024-09-11T09:15:11.414779+0000","last_fullsized":"2024-09-11T09:15:11.414779+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:22:21.610509+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":914,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.415011+0000","last_change":"2024-09-11T08:39:27.015272+0000","last_active":"2024-09-11T09:15:11.415011+0000","last_peered":"2024-09-11T09:15:11.415011+0000","last_clean":"2024-09-11T09:15:11.415011+0000","last_became_active":"2024-09-11T08:39:24.995216+0000","last_became_peered":"2024-09-11T08:39:24.995216+0000","last_unstale":"2024-09-11T09:15:11.415011+0000","last_undegraded":"2024-09-11T09:15:11.415011+0000","last_fullsized":"2024-09-11T09:15:11.415011+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:45:49.872594+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.414856+0000","last_change":"2024-09-11T08:39:27.014087+0000","last_active":"2024-09-11T09:15:11.414856+0000","last_peered":"2024-09-11T09:15:11.414856+0000","last_clean":"2024-09-11T09:15:11.414856+0000","last_became_active":"2024-09-11T08:39:24.994002+0000","last_became_peered":"2024-09-11T08:39:24.994002+0000","last_unstale":"2024-09-11T09:15:11.414856+0000","last_undegraded":"2024-09-11T09:15:11.414856+0000","last_fullsized":"2024-09-11T09:15:11.414856+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:54:18.729378+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070708900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":915,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.417014+0000","last_change":"2024-09-11T08:39:27.014269+0000","last_active":"2024-09-11T09:15:11.417014+0000","last_peered":"2024-09-11T09:15:11.417014+0000","last_clean":"2024-09-11T09:15:11.417014+0000","last_became_active":"2024-09-11T08:39:24.995202+0000","last_became_peered":"2024-09-11T08:39:24.995202+0000","last_unstale":"2024-09-11T09:15:11.417014+0000","last_undegraded":"2024-09-11T09:15:11.417014+0000","last_fullsized":"2024-09-11T09:15:11.417014+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:55:38.241377+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010507380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.395116+0000","last_change":"2024-09-11T08:39:27.008400+0000","last_active":"2024-09-11T09:15:11.395116+0000","last_peered":"2024-09-11T09:15:11.395116+0000","last_clean":"2024-09-11T09:15:11.395116+0000","last_became_active":"2024-09-11T08:39:24.990422+0000","last_became_peered":"2024-09-11T08:39:24.990422+0000","last_unstale":"2024-09-11T09:15:11.395116+0000","last_undegraded":"2024-09-11T09:15:11.395116+0000","last_fullsized":"2024-09-11T09:15:11.395116+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T08:59:33.107916+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000934803,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.960990+0000","last_change":"2024-09-11T08:39:27.657840+0000","last_active":"2024-09-11T09:15:10.960990+0000","last_peered":"2024-09-11T09:15:10.960990+0000","last_clean":"2024-09-11T09:15:10.960990+0000","last_became_active":"2024-09-11T08:39:24.994237+0000","last_became_peered":"2024-09-11T08:39:24.994237+0000","last_unstale":"2024-09-11T09:15:10.960990+0000","last_undegraded":"2024-09-11T09:15:10.960990+0000","last_fullsized":"2024-09-11T09:15:10.960990+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:25:53.237878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063005199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.962738+0000","last_change":"2024-09-11T08:39:27.657633+0000","last_active":"2024-09-11T09:15:10.962738+0000","last_peered":"2024-09-11T09:15:10.962738+0000","last_clean":"2024-09-11T09:15:10.962738+0000","last_became_active":"2024-09-11T08:39:24.990604+0000","last_became_peered":"2024-09-11T08:39:24.990604+0000","last_unstale":"2024-09-11T09:15:10.962738+0000","last_undegraded":"2024-09-11T09:15:10.962738+0000","last_fullsized":"2024-09-11T09:15:10.962738+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:42:38.112533+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000543883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":916,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416251+0000","last_change":"2024-09-11T08:39:27.015047+0000","last_active":"2024-09-11T09:15:11.416251+0000","last_peered":"2024-09-11T09:15:11.416251+0000","last_clean":"2024-09-11T09:15:11.416251+0000","last_became_active":"2024-09-11T08:39:24.987701+0000","last_became_peered":"2024-09-11T08:39:24.987701+0000","last_unstale":"2024-09-11T09:15:11.416251+0000","last_undegraded":"2024-09-11T09:15:11.416251+0000","last_fullsized":"2024-09-11T09:15:11.416251+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:23.968890+0000","last_clean_scrub_stamp":"2024-09-11T08:39:23.968890+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:55:50.605878+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016047609999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"10'97","reported_seq":1042,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.416498+0000","last_change":"2024-09-11T08:39:22.547555+0000","last_active":"2024-09-11T09:15:11.416498+0000","last_peered":"2024-09-11T09:15:11.416498+0000","last_clean":"2024-09-11T09:15:11.416498+0000","last_became_active":"2024-09-11T08:39:22.544974+0000","last_became_peered":"2024-09-11T08:39:22.544974+0000","last_unstale":"2024-09-11T09:15:11.416498+0000","last_undegraded":"2024-09-11T09:15:11.416498+0000","last_fullsized":"2024-09-11T09:15:11.416498+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:39:20.961806+0000","last_clean_scrub_stamp":"2024-09-11T08:39:20.961806+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:35:01.190641+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738804,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838952,"kb_used_data":5264,"kb_used_omap":101,"kb_used_meta":833562,"kb_avail":93532888,"statfs":{"total":96636764160,"available":95777677312,"internally_reserved":0,"allocated":5390336,"data_stored":2639220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103937,"internal_metadata":853567999},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738806,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717356,"kb_used_data":5848,"kb_used_omap":95,"kb_used_meta":711392,"kb_avail":93654484,"statfs":{"total":96636764160,"available":95902191616,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97785,"internal_metadata":728465927},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":1,"apply_latency_ms":1,"commit_latency_ns":1000000,"apply_latency_ns":1000000},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738804,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1123756,"kb_used_data":5848,"kb_used_omap":105,"kb_used_meta":1117782,"kb_avail":93248084,"statfs":{"total":96636764160,"available":95486038016,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107949,"internal_metadata":1144609363},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.416 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T08:39:23.968890+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=23, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.417 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T08:39:20.961806+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=39, tm_sec=20, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=15, tm_sec=19, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:15:22.417 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T09:15:42.418 DEBUG:teuthology.orchestra.run.smithi077:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster2 pg dump --format=json 2024-09-11T09:15:42.729 INFO:teuthology.orchestra.run.smithi077.stdout: 2024-09-11T09:15:42.730 INFO:teuthology.orchestra.run.smithi077.stderr:dumped all 2024-09-11T09:15:42.748 INFO:teuthology.orchestra.run.smithi077.stdout:{"pg_ready":true,"pg_map":{"version":1543,"stamp":"2024-09-11T09:15:40.791426+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":2680048,"kb_used_data":16960,"kb_used_omap":302,"kb_used_meta":2662737,"kb_avail":280435472,"statfs":{"total":289910292480,"available":287165923328,"internally_reserved":0,"allocated":17367040,"data_stored":9098434,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":309671,"internal_metadata":2726643289},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.003316"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":923,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:24.127360+0000","last_change":"2024-09-11T09:15:24.127413+0000","last_active":"2024-09-11T09:15:24.127360+0000","last_peered":"2024-09-11T09:15:24.127360+0000","last_clean":"2024-09-11T09:15:24.127360+0000","last_became_active":"2024-09-11T08:39:24.993902+0000","last_became_peered":"2024-09-11T08:39:24.993902+0000","last_unstale":"2024-09-11T09:15:24.127360+0000","last_undegraded":"2024-09-11T09:15:24.127360+0000","last_fullsized":"2024-09-11T09:15:24.127360+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:15:24.127178+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:15:24.127178+0000","last_clean_scrub_stamp":"2024-09-11T09:15:24.127178+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:33:47.151121+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001275038,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":923,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:26.155994+0000","last_change":"2024-09-11T09:15:26.156049+0000","last_active":"2024-09-11T09:15:26.155994+0000","last_peered":"2024-09-11T09:15:26.155994+0000","last_clean":"2024-09-11T09:15:26.155994+0000","last_became_active":"2024-09-11T08:39:24.995216+0000","last_became_peered":"2024-09-11T08:39:24.995216+0000","last_unstale":"2024-09-11T09:15:26.155994+0000","last_undegraded":"2024-09-11T09:15:26.155994+0000","last_fullsized":"2024-09-11T09:15:26.155994+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:15:26.155793+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:15:26.155793+0000","last_clean_scrub_stamp":"2024-09-11T09:15:26.155793+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:16:52.136572+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0014367659999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":924,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:25.173992+0000","last_change":"2024-09-11T09:15:25.174056+0000","last_active":"2024-09-11T09:15:25.173992+0000","last_peered":"2024-09-11T09:15:25.173992+0000","last_clean":"2024-09-11T09:15:25.173992+0000","last_became_active":"2024-09-11T08:39:24.994002+0000","last_became_peered":"2024-09-11T08:39:24.994002+0000","last_unstale":"2024-09-11T09:15:25.173992+0000","last_undegraded":"2024-09-11T09:15:25.173992+0000","last_fullsized":"2024-09-11T09:15:25.173992+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:15:25.173772+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:15:25.173772+0000","last_clean_scrub_stamp":"2024-09-11T09:15:25.173772+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T18:33:40.752496+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00070708900000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":924,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:27.121624+0000","last_change":"2024-09-11T09:15:27.121680+0000","last_active":"2024-09-11T09:15:27.121624+0000","last_peered":"2024-09-11T09:15:27.121624+0000","last_clean":"2024-09-11T09:15:27.121624+0000","last_became_active":"2024-09-11T08:39:24.995202+0000","last_became_peered":"2024-09-11T08:39:24.995202+0000","last_unstale":"2024-09-11T09:15:27.121624+0000","last_undegraded":"2024-09-11T09:15:27.121624+0000","last_fullsized":"2024-09-11T09:15:27.121624+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:15:27.121420+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:15:27.121420+0000","last_clean_scrub_stamp":"2024-09-11T09:15:27.121420+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:23:53.444480+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010507380000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"455'3","reported_seq":926,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:21.952571+0000","last_change":"2024-09-11T09:15:21.952608+0000","last_active":"2024-09-11T09:15:21.952571+0000","last_peered":"2024-09-11T09:15:21.952571+0000","last_clean":"2024-09-11T09:15:21.952571+0000","last_became_active":"2024-09-11T08:39:24.990422+0000","last_became_peered":"2024-09-11T08:39:24.990422+0000","last_unstale":"2024-09-11T09:15:21.952571+0000","last_undegraded":"2024-09-11T09:15:21.952571+0000","last_fullsized":"2024-09-11T09:15:21.952571+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'3","last_scrub_stamp":"2024-09-11T09:15:21.952485+0000","last_deep_scrub":"455'3","last_deep_scrub_stamp":"2024-09-11T09:15:21.952485+0000","last_clean_scrub_stamp":"2024-09-11T09:15:21.952485+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:35:15.451552+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000934803,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":925,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:23.856032+0000","last_change":"2024-09-11T09:15:23.856086+0000","last_active":"2024-09-11T09:15:23.856032+0000","last_peered":"2024-09-11T09:15:23.856032+0000","last_clean":"2024-09-11T09:15:23.856032+0000","last_became_active":"2024-09-11T08:39:24.994237+0000","last_became_peered":"2024-09-11T08:39:24.994237+0000","last_unstale":"2024-09-11T09:15:23.856032+0000","last_undegraded":"2024-09-11T09:15:23.856032+0000","last_fullsized":"2024-09-11T09:15:23.856032+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:15:23.855841+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:15:23.855841+0000","last_clean_scrub_stamp":"2024-09-11T09:15:23.855841+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:10:12.272909+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00063005199999999998,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":926,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:22.878883+0000","last_change":"2024-09-11T09:15:22.878883+0000","last_active":"2024-09-11T09:15:22.878883+0000","last_peered":"2024-09-11T09:15:22.878883+0000","last_clean":"2024-09-11T09:15:22.878883+0000","last_became_active":"2024-09-11T08:39:24.990604+0000","last_became_peered":"2024-09-11T08:39:24.990604+0000","last_unstale":"2024-09-11T09:15:22.878883+0000","last_undegraded":"2024-09-11T09:15:22.878883+0000","last_fullsized":"2024-09-11T09:15:22.878883+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:15:22.878623+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:15:22.878623+0000","last_clean_scrub_stamp":"2024-09-11T09:15:22.878623+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:51:00.374722+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.000543883,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"455'2","reported_seq":926,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:23.084989+0000","last_change":"2024-09-11T09:15:23.085057+0000","last_active":"2024-09-11T09:15:23.084989+0000","last_peered":"2024-09-11T09:15:23.084989+0000","last_clean":"2024-09-11T09:15:23.084989+0000","last_became_active":"2024-09-11T08:39:24.987701+0000","last_became_peered":"2024-09-11T08:39:24.987701+0000","last_unstale":"2024-09-11T09:15:23.084989+0000","last_undegraded":"2024-09-11T09:15:23.084989+0000","last_fullsized":"2024-09-11T09:15:23.084989+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'2","last_scrub_stamp":"2024-09-11T09:15:23.084802+0000","last_deep_scrub":"455'2","last_deep_scrub_stamp":"2024-09-11T09:15:23.084802+0000","last_clean_scrub_stamp":"2024-09-11T09:15:23.084802+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:31:14.475846+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0.0016047609999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"455'99","reported_seq":1054,"reported_epoch":455,"state":"active+clean","last_fresh":"2024-09-11T09:15:22.086707+0000","last_change":"2024-09-11T09:15:22.086707+0000","last_active":"2024-09-11T09:15:22.086707+0000","last_peered":"2024-09-11T09:15:22.086707+0000","last_clean":"2024-09-11T09:15:22.086707+0000","last_became_active":"2024-09-11T08:39:22.544974+0000","last_became_peered":"2024-09-11T08:39:22.544974+0000","last_unstale":"2024-09-11T09:15:22.086707+0000","last_undegraded":"2024-09-11T09:15:22.086707+0000","last_fullsized":"2024-09-11T09:15:22.086707+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"455'99","last_scrub_stamp":"2024-09-11T09:15:22.086358+0000","last_deep_scrub":"455'99","last_deep_scrub_stamp":"2024-09-11T09:15:22.086358+0000","last_clean_scrub_stamp":"2024-09-11T09:15:22.086358+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T15:44:36.281159+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738808,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":838944,"kb_used_data":5264,"kb_used_omap":101,"kb_used_meta":833562,"kb_avail":93532896,"statfs":{"total":96636764160,"available":95777685504,"internally_reserved":0,"allocated":5390336,"data_stored":2639220,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":103937,"internal_metadata":853567999},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738810,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":717352,"kb_used_data":5848,"kb_used_omap":95,"kb_used_meta":711392,"kb_avail":93654488,"statfs":{"total":96636764160,"available":95902195712,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":97785,"internal_metadata":728465927},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738808,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":1123752,"kb_used_data":5848,"kb_used_omap":105,"kb_used_meta":1117782,"kb_avail":93248088,"statfs":{"total":96636764160,"available":95486042112,"internally_reserved":0,"allocated":5988352,"data_stored":3229607,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":107949,"internal_metadata":1144609363},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:15:42.749 DEBUG:teuthology.orchestra.run.smithi077:> sudo ceph --cluster cluster2 config set global mon_health_to_clog false 2024-09-11T09:15:43.072 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T09:15:43.072 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T09:15:43.072 DEBUG:tasks.ceph.cluster2.osd.0:waiting for process to exit 2024-09-11T09:15:43.072 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:47.120 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 is failed for ~0s 2024-09-11T09:15:49.174 INFO:tasks.ceph.cluster2.osd.0:Stopped 2024-09-11T09:15:49.174 DEBUG:tasks.ceph.cluster2.osd.1:waiting for process to exit 2024-09-11T09:15:49.174 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:52.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 is failed for ~0s 2024-09-11T09:15:52.423 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.0 has been restored 2024-09-11T09:15:55.276 INFO:tasks.ceph.cluster2.osd.1:Stopped 2024-09-11T09:15:55.276 DEBUG:tasks.ceph.cluster2.osd.2:waiting for process to exit 2024-09-11T09:15:55.276 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:15:57.625 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 is failed for ~0s 2024-09-11T09:15:57.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.1 has been restored 2024-09-11T09:16:01.378 INFO:tasks.ceph.cluster2.osd.2:Stopped 2024-09-11T09:16:01.378 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T09:16:01.379 DEBUG:tasks.ceph.cluster2.mgr.x:waiting for process to exit 2024-09-11T09:16:01.379 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:16:01.408 INFO:tasks.ceph.cluster2.mgr.x:Stopped 2024-09-11T09:16:01.408 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T09:16:01.408 DEBUG:tasks.ceph.cluster2.mon.a:waiting for process to exit 2024-09-11T09:16:01.408 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:16:01.432 INFO:tasks.ceph.cluster2.mon.a:Stopped 2024-09-11T09:16:01.432 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T09:16:01.433 DEBUG:teuthology.orchestra.run.smithi077:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster2.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T09:16:01.492 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-0 on ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T09:16:01.492 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-0 2024-09-11T09:16:01.838 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-1 on ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T09:16:01.839 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-1 2024-09-11T09:16:02.114 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster2-2 on ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T09:16:02.114 DEBUG:teuthology.orchestra.run.smithi077:> sync && sudo umount -f /var/lib/ceph/osd/cluster2-2 2024-09-11T09:16:02.505 INFO:tasks.ceph:Archiving mon data... 2024-09-11T09:16:02.507 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/mon/cluster2-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/data/cluster2.mon.a.tgz 2024-09-11T09:16:02.507 DEBUG:teuthology.orchestra.run.smithi077:> mktemp 2024-09-11T09:16:02.512 INFO:teuthology.orchestra.run.smithi077.stdout:/tmp/tmp.0cSSKmxddO 2024-09-11T09:16:02.512 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster2-a -- . > /tmp/tmp.0cSSKmxddO 2024-09-11T09:16:02.626 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster2.osd.2 has been restored 2024-09-11T09:16:02.670 DEBUG:teuthology.orchestra.run.smithi077:> sudo chmod 0666 /tmp/tmp.0cSSKmxddO 2024-09-11T09:16:02.732 DEBUG:teuthology.orchestra.remote:smithi077:/tmp/tmp.0cSSKmxddO is 460KB 2024-09-11T09:16:02.794 DEBUG:teuthology.orchestra.run.smithi077:> rm -fr /tmp/tmp.0cSSKmxddO 2024-09-11T09:16:02.799 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T09:16:02.799 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T09:16:02.802 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/cluster2.conf /etc/ceph/cluster2.keyring /home/ubuntu/cephtest/cluster2.data /home/ubuntu/cephtest/cluster2.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T09:16:02.852 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2024-09-11T09:16:02.864 INFO:tasks.ceph.ceph_manager.cluster1:waiting for clean 2024-09-11T09:16:02.864 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T09:16:03.170 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T09:16:03.170 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-09-11T09:16:03.189 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-11T09:16:02.239769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":300240,"kb_used_data":12496,"kb_used_omap":4,"kb_used_meta":287675,"kb_avail":282815280,"statfs":{"total":289910292480,"available":289602846720,"internally_reserved":0,"allocated":12795904,"data_stored":7384639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":294579535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002980"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559585+0000","last_change":"2024-09-11T08:38:47.626867+0000","last_active":"2024-09-11T09:15:11.559585+0000","last_peered":"2024-09-11T09:15:11.559585+0000","last_clean":"2024-09-11T09:15:11.559585+0000","last_became_active":"2024-09-11T08:38:45.609230+0000","last_became_peered":"2024-09-11T08:38:45.609230+0000","last_unstale":"2024-09-11T09:15:11.559585+0000","last_undegraded":"2024-09-11T09:15:11.559585+0000","last_fullsized":"2024-09-11T09:15:11.559585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:39:06.699918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021686650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559681+0000","last_change":"2024-09-11T08:38:47.626289+0000","last_active":"2024-09-11T09:15:11.559681+0000","last_peered":"2024-09-11T09:15:11.559681+0000","last_clean":"2024-09-11T09:15:11.559681+0000","last_became_active":"2024-09-11T08:38:45.609290+0000","last_became_peered":"2024-09-11T08:38:45.609290+0000","last_unstale":"2024-09-11T09:15:11.559681+0000","last_undegraded":"2024-09-11T09:15:11.559681+0000","last_fullsized":"2024-09-11T09:15:11.559681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:54.913852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605272,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559870+0000","last_change":"2024-09-11T08:38:47.625598+0000","last_active":"2024-09-11T09:15:11.559870+0000","last_peered":"2024-09-11T09:15:11.559870+0000","last_clean":"2024-09-11T09:15:11.559870+0000","last_became_active":"2024-09-11T08:38:45.606786+0000","last_became_peered":"2024-09-11T08:38:45.606786+0000","last_unstale":"2024-09-11T09:15:11.559870+0000","last_undegraded":"2024-09-11T09:15:11.559870+0000","last_fullsized":"2024-09-11T09:15:11.559870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:01:02.286892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098543599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559911+0000","last_change":"2024-09-11T08:38:47.625546+0000","last_active":"2024-09-11T09:15:11.559911+0000","last_peered":"2024-09-11T09:15:11.559911+0000","last_clean":"2024-09-11T09:15:11.559911+0000","last_became_active":"2024-09-11T08:38:45.606856+0000","last_became_peered":"2024-09-11T08:38:45.606856+0000","last_unstale":"2024-09-11T09:15:11.559911+0000","last_undegraded":"2024-09-11T09:15:11.559911+0000","last_fullsized":"2024-09-11T09:15:11.559911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:04:07.641669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098068000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":948,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.145247+0000","last_change":"2024-09-11T08:38:47.622095+0000","last_active":"2024-09-11T09:15:10.145247+0000","last_peered":"2024-09-11T09:15:10.145247+0000","last_clean":"2024-09-11T09:15:10.145247+0000","last_became_active":"2024-09-11T08:38:45.603681+0000","last_became_peered":"2024-09-11T08:38:45.603681+0000","last_unstale":"2024-09-11T09:15:10.145247+0000","last_undegraded":"2024-09-11T09:15:10.145247+0000","last_fullsized":"2024-09-11T09:15:10.145247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:20:16.311770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021015780000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":944,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.456071+0000","last_change":"2024-09-11T08:38:47.615033+0000","last_active":"2024-09-11T09:15:10.456071+0000","last_peered":"2024-09-11T09:15:10.456071+0000","last_clean":"2024-09-11T09:15:10.456071+0000","last_became_active":"2024-09-11T08:38:45.604617+0000","last_became_peered":"2024-09-11T08:38:45.604617+0000","last_unstale":"2024-09-11T09:15:10.456071+0000","last_undegraded":"2024-09-11T09:15:10.456071+0000","last_fullsized":"2024-09-11T09:15:10.456071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:30.831517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010008059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.447746+0000","last_change":"2024-09-11T08:38:47.615034+0000","last_active":"2024-09-11T09:15:10.447746+0000","last_peered":"2024-09-11T09:15:10.447746+0000","last_clean":"2024-09-11T09:15:10.447746+0000","last_became_active":"2024-09-11T08:38:45.600541+0000","last_became_peered":"2024-09-11T08:38:45.600541+0000","last_unstale":"2024-09-11T09:15:10.447746+0000","last_undegraded":"2024-09-11T09:15:10.447746+0000","last_fullsized":"2024-09-11T09:15:10.447746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:46.935084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":946,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559181+0000","last_change":"2024-09-11T08:38:47.626283+0000","last_active":"2024-09-11T09:15:11.559181+0000","last_peered":"2024-09-11T09:15:11.559181+0000","last_clean":"2024-09-11T09:15:11.559181+0000","last_became_active":"2024-09-11T08:38:45.605202+0000","last_became_peered":"2024-09-11T08:38:45.605202+0000","last_unstale":"2024-09-11T09:15:11.559181+0000","last_undegraded":"2024-09-11T09:15:11.559181+0000","last_fullsized":"2024-09-11T09:15:11.559181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:51:20.908060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016448249999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1074,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.561308+0000","last_change":"2024-09-11T08:38:43.315778+0000","last_active":"2024-09-11T09:15:11.561308+0000","last_peered":"2024-09-11T09:15:11.561308+0000","last_clean":"2024-09-11T09:15:11.561308+0000","last_became_active":"2024-09-11T08:38:43.314266+0000","last_became_peered":"2024-09-11T08:38:43.314266+0000","last_unstale":"2024-09-11T09:15:11.561308+0000","last_undegraded":"2024-09-11T09:15:11.561308+0000","last_fullsized":"2024-09-11T09:15:11.561308+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_clean_scrub_stamp":"2024-09-11T08:38:41.492511+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:19.438131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89044,"kb_used_data":3776,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282796,"statfs":{"total":96636764160,"available":96545583104,"internally_reserved":0,"allocated":3866624,"data_stored":2067955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89628,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282212,"statfs":{"total":96636764160,"available":96544985088,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738821,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121568,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250272,"statfs":{"total":96636764160,"available":96512278528,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:16:03.190 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T09:16:03.485 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T09:16:03.485 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-09-11T09:16:03.504 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-11T09:16:02.239769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":300240,"kb_used_data":12496,"kb_used_omap":4,"kb_used_meta":287675,"kb_avail":282815280,"statfs":{"total":289910292480,"available":289602846720,"internally_reserved":0,"allocated":12795904,"data_stored":7384639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":294579535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002980"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559585+0000","last_change":"2024-09-11T08:38:47.626867+0000","last_active":"2024-09-11T09:15:11.559585+0000","last_peered":"2024-09-11T09:15:11.559585+0000","last_clean":"2024-09-11T09:15:11.559585+0000","last_became_active":"2024-09-11T08:38:45.609230+0000","last_became_peered":"2024-09-11T08:38:45.609230+0000","last_unstale":"2024-09-11T09:15:11.559585+0000","last_undegraded":"2024-09-11T09:15:11.559585+0000","last_fullsized":"2024-09-11T09:15:11.559585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:39:06.699918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021686650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559681+0000","last_change":"2024-09-11T08:38:47.626289+0000","last_active":"2024-09-11T09:15:11.559681+0000","last_peered":"2024-09-11T09:15:11.559681+0000","last_clean":"2024-09-11T09:15:11.559681+0000","last_became_active":"2024-09-11T08:38:45.609290+0000","last_became_peered":"2024-09-11T08:38:45.609290+0000","last_unstale":"2024-09-11T09:15:11.559681+0000","last_undegraded":"2024-09-11T09:15:11.559681+0000","last_fullsized":"2024-09-11T09:15:11.559681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:54.913852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605272,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559870+0000","last_change":"2024-09-11T08:38:47.625598+0000","last_active":"2024-09-11T09:15:11.559870+0000","last_peered":"2024-09-11T09:15:11.559870+0000","last_clean":"2024-09-11T09:15:11.559870+0000","last_became_active":"2024-09-11T08:38:45.606786+0000","last_became_peered":"2024-09-11T08:38:45.606786+0000","last_unstale":"2024-09-11T09:15:11.559870+0000","last_undegraded":"2024-09-11T09:15:11.559870+0000","last_fullsized":"2024-09-11T09:15:11.559870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:01:02.286892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098543599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559911+0000","last_change":"2024-09-11T08:38:47.625546+0000","last_active":"2024-09-11T09:15:11.559911+0000","last_peered":"2024-09-11T09:15:11.559911+0000","last_clean":"2024-09-11T09:15:11.559911+0000","last_became_active":"2024-09-11T08:38:45.606856+0000","last_became_peered":"2024-09-11T08:38:45.606856+0000","last_unstale":"2024-09-11T09:15:11.559911+0000","last_undegraded":"2024-09-11T09:15:11.559911+0000","last_fullsized":"2024-09-11T09:15:11.559911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:04:07.641669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098068000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":948,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.145247+0000","last_change":"2024-09-11T08:38:47.622095+0000","last_active":"2024-09-11T09:15:10.145247+0000","last_peered":"2024-09-11T09:15:10.145247+0000","last_clean":"2024-09-11T09:15:10.145247+0000","last_became_active":"2024-09-11T08:38:45.603681+0000","last_became_peered":"2024-09-11T08:38:45.603681+0000","last_unstale":"2024-09-11T09:15:10.145247+0000","last_undegraded":"2024-09-11T09:15:10.145247+0000","last_fullsized":"2024-09-11T09:15:10.145247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:20:16.311770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021015780000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":944,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.456071+0000","last_change":"2024-09-11T08:38:47.615033+0000","last_active":"2024-09-11T09:15:10.456071+0000","last_peered":"2024-09-11T09:15:10.456071+0000","last_clean":"2024-09-11T09:15:10.456071+0000","last_became_active":"2024-09-11T08:38:45.604617+0000","last_became_peered":"2024-09-11T08:38:45.604617+0000","last_unstale":"2024-09-11T09:15:10.456071+0000","last_undegraded":"2024-09-11T09:15:10.456071+0000","last_fullsized":"2024-09-11T09:15:10.456071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:30.831517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010008059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.447746+0000","last_change":"2024-09-11T08:38:47.615034+0000","last_active":"2024-09-11T09:15:10.447746+0000","last_peered":"2024-09-11T09:15:10.447746+0000","last_clean":"2024-09-11T09:15:10.447746+0000","last_became_active":"2024-09-11T08:38:45.600541+0000","last_became_peered":"2024-09-11T08:38:45.600541+0000","last_unstale":"2024-09-11T09:15:10.447746+0000","last_undegraded":"2024-09-11T09:15:10.447746+0000","last_fullsized":"2024-09-11T09:15:10.447746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:46.935084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":946,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559181+0000","last_change":"2024-09-11T08:38:47.626283+0000","last_active":"2024-09-11T09:15:11.559181+0000","last_peered":"2024-09-11T09:15:11.559181+0000","last_clean":"2024-09-11T09:15:11.559181+0000","last_became_active":"2024-09-11T08:38:45.605202+0000","last_became_peered":"2024-09-11T08:38:45.605202+0000","last_unstale":"2024-09-11T09:15:11.559181+0000","last_undegraded":"2024-09-11T09:15:11.559181+0000","last_fullsized":"2024-09-11T09:15:11.559181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:51:20.908060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016448249999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1074,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.561308+0000","last_change":"2024-09-11T08:38:43.315778+0000","last_active":"2024-09-11T09:15:11.561308+0000","last_peered":"2024-09-11T09:15:11.561308+0000","last_clean":"2024-09-11T09:15:11.561308+0000","last_became_active":"2024-09-11T08:38:43.314266+0000","last_became_peered":"2024-09-11T08:38:43.314266+0000","last_unstale":"2024-09-11T09:15:11.561308+0000","last_undegraded":"2024-09-11T09:15:11.561308+0000","last_fullsized":"2024-09-11T09:15:11.561308+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_clean_scrub_stamp":"2024-09-11T08:38:41.492511+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:19.438131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89044,"kb_used_data":3776,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282796,"statfs":{"total":96636764160,"available":96545583104,"internally_reserved":0,"allocated":3866624,"data_stored":2067955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89628,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282212,"statfs":{"total":96636764160,"available":96544985088,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738821,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121568,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250272,"statfs":{"total":96636764160,"available":96512278528,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:16:03.505 INFO:tasks.ceph.ceph_manager.cluster1:clean! 2024-09-11T09:16:03.505 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T09:16:03.796 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T09:16:03.796 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-09-11T09:16:03.814 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":1589,"stamp":"2024-09-11T09:16:02.239769+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":100,"ondisk_log_size":100,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":300240,"kb_used_data":12496,"kb_used_omap":4,"kb_used_meta":287675,"kb_avail":282815280,"statfs":{"total":289910292480,"available":289602846720,"internally_reserved":0,"allocated":12795904,"data_stored":7384639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":294579535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002980"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559585+0000","last_change":"2024-09-11T08:38:47.626867+0000","last_active":"2024-09-11T09:15:11.559585+0000","last_peered":"2024-09-11T09:15:11.559585+0000","last_clean":"2024-09-11T09:15:11.559585+0000","last_became_active":"2024-09-11T08:38:45.609230+0000","last_became_peered":"2024-09-11T08:38:45.609230+0000","last_unstale":"2024-09-11T09:15:11.559585+0000","last_undegraded":"2024-09-11T09:15:11.559585+0000","last_fullsized":"2024-09-11T09:15:11.559585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:39:06.699918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021686650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559681+0000","last_change":"2024-09-11T08:38:47.626289+0000","last_active":"2024-09-11T09:15:11.559681+0000","last_peered":"2024-09-11T09:15:11.559681+0000","last_clean":"2024-09-11T09:15:11.559681+0000","last_became_active":"2024-09-11T08:38:45.609290+0000","last_became_peered":"2024-09-11T08:38:45.609290+0000","last_unstale":"2024-09-11T09:15:11.559681+0000","last_undegraded":"2024-09-11T09:15:11.559681+0000","last_fullsized":"2024-09-11T09:15:11.559681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:54.913852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605272,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559870+0000","last_change":"2024-09-11T08:38:47.625598+0000","last_active":"2024-09-11T09:15:11.559870+0000","last_peered":"2024-09-11T09:15:11.559870+0000","last_clean":"2024-09-11T09:15:11.559870+0000","last_became_active":"2024-09-11T08:38:45.606786+0000","last_became_peered":"2024-09-11T08:38:45.606786+0000","last_unstale":"2024-09-11T09:15:11.559870+0000","last_undegraded":"2024-09-11T09:15:11.559870+0000","last_fullsized":"2024-09-11T09:15:11.559870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:01:02.286892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098543599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559911+0000","last_change":"2024-09-11T08:38:47.625546+0000","last_active":"2024-09-11T09:15:11.559911+0000","last_peered":"2024-09-11T09:15:11.559911+0000","last_clean":"2024-09-11T09:15:11.559911+0000","last_became_active":"2024-09-11T08:38:45.606856+0000","last_became_peered":"2024-09-11T08:38:45.606856+0000","last_unstale":"2024-09-11T09:15:11.559911+0000","last_undegraded":"2024-09-11T09:15:11.559911+0000","last_fullsized":"2024-09-11T09:15:11.559911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:04:07.641669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098068000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"15'2","reported_seq":948,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.145247+0000","last_change":"2024-09-11T08:38:47.622095+0000","last_active":"2024-09-11T09:15:10.145247+0000","last_peered":"2024-09-11T09:15:10.145247+0000","last_clean":"2024-09-11T09:15:10.145247+0000","last_became_active":"2024-09-11T08:38:45.603681+0000","last_became_peered":"2024-09-11T08:38:45.603681+0000","last_unstale":"2024-09-11T09:15:10.145247+0000","last_undegraded":"2024-09-11T09:15:10.145247+0000","last_fullsized":"2024-09-11T09:15:10.145247+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T11:20:16.311770+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021015780000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":944,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.456071+0000","last_change":"2024-09-11T08:38:47.615033+0000","last_active":"2024-09-11T09:15:10.456071+0000","last_peered":"2024-09-11T09:15:10.456071+0000","last_clean":"2024-09-11T09:15:10.456071+0000","last_became_active":"2024-09-11T08:38:45.604617+0000","last_became_peered":"2024-09-11T08:38:45.604617+0000","last_unstale":"2024-09-11T09:15:10.456071+0000","last_undegraded":"2024-09-11T09:15:10.456071+0000","last_fullsized":"2024-09-11T09:15:10.456071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:30.831517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010008059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.447746+0000","last_change":"2024-09-11T08:38:47.615034+0000","last_active":"2024-09-11T09:15:10.447746+0000","last_peered":"2024-09-11T09:15:10.447746+0000","last_clean":"2024-09-11T09:15:10.447746+0000","last_became_active":"2024-09-11T08:38:45.600541+0000","last_became_peered":"2024-09-11T08:38:45.600541+0000","last_unstale":"2024-09-11T09:15:10.447746+0000","last_undegraded":"2024-09-11T09:15:10.447746+0000","last_fullsized":"2024-09-11T09:15:10.447746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:46.935084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":946,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559181+0000","last_change":"2024-09-11T08:38:47.626283+0000","last_active":"2024-09-11T09:15:11.559181+0000","last_peered":"2024-09-11T09:15:11.559181+0000","last_clean":"2024-09-11T09:15:11.559181+0000","last_became_active":"2024-09-11T08:38:45.605202+0000","last_became_peered":"2024-09-11T08:38:45.605202+0000","last_unstale":"2024-09-11T09:15:11.559181+0000","last_undegraded":"2024-09-11T09:15:11.559181+0000","last_fullsized":"2024-09-11T09:15:11.559181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:51:20.908060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016448249999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1074,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.561308+0000","last_change":"2024-09-11T08:38:43.315778+0000","last_active":"2024-09-11T09:15:11.561308+0000","last_peered":"2024-09-11T09:15:11.561308+0000","last_clean":"2024-09-11T09:15:11.561308+0000","last_became_active":"2024-09-11T08:38:43.314266+0000","last_became_peered":"2024-09-11T08:38:43.314266+0000","last_unstale":"2024-09-11T09:15:11.561308+0000","last_undegraded":"2024-09-11T09:15:11.561308+0000","last_fullsized":"2024-09-11T09:15:11.561308+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_clean_scrub_stamp":"2024-09-11T08:38:41.492511+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:19.438131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":3,"ondisk_log_size":3,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738821,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89044,"kb_used_data":3776,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282796,"statfs":{"total":96636764160,"available":96545583104,"internally_reserved":0,"allocated":3866624,"data_stored":2067955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738820,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89628,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282212,"statfs":{"total":96636764160,"available":96544985088,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738821,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121568,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250272,"statfs":{"total":96636764160,"available":96512278528,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:16:03.815 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd dump --format=json 2024-09-11T09:16:04.109 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T09:16:04.109 INFO:teuthology.orchestra.run.smithi073.stdout:{"epoch":470,"fsid":"9661569a-ff40-4cdb-a5be-7b937214efd1","created":"2024-09-11T08:38:28.689051+0000","modified":"2024-09-11T09:15:10.126316+0000","last_up_change":"2024-09-11T08:38:40.431031+0000","last_in_change":"2024-09-11T08:38:29.621519+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":3,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"squid","allow_crimson":false,"pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-09-11T08:38:41.491826+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"11","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}},"read_balance":{"score_type":"Fair distribution","score_acting":2.9900000095367432,"score_stable":2.9900000095367432,"optimal_score":0.67000001668930054,"raw_score_acting":2,"raw_score_stable":2,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}},{"pool":2,"pool_name":"rbd","create_time":"2024-09-11T08:38:43.945430+0000","flags":8193,"flags_names":"hashpspool,selfmanaged_snaps","type":1,"size":2,"min_size":1,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"is_stretch_pool":false,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":2,"snap_epoch":15,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}},"read_balance":{"score_type":"Fair distribution","score_acting":1.8799999952316284,"score_stable":1.8799999952316284,"optimal_score":1,"raw_score_acting":1.8799999952316284,"raw_score_stable":1.8799999952316284,"primary_affinity_weighted":1,"average_primary_affinity":1,"average_primary_affinity_weighted":1}}],"osds":[{"osd":0,"uuid":"9413ccf2-d186-4993-a666-0cb76bc6a459","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6816","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6817","nonce":2968504120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6818","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6819","nonce":2968504120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6822","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6823","nonce":2968504120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6820","nonce":2968504120},{"type":"v1","addr":"172.21.15.73:6821","nonce":2968504120}]},"public_addr":"172.21.15.73:6817/2968504120","cluster_addr":"172.21.15.73:6819/2968504120","heartbeat_back_addr":"172.21.15.73:6823/2968504120","heartbeat_front_addr":"172.21.15.73:6821/2968504120","state":["exists","up"]},{"osd":1,"uuid":"58e98484-e7c9-412c-821d-5f3d8ea52733","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6808","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6809","nonce":56303921}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6810","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6811","nonce":56303921}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6814","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6815","nonce":56303921}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6812","nonce":56303921},{"type":"v1","addr":"172.21.15.73:6813","nonce":56303921}]},"public_addr":"172.21.15.73:6809/56303921","cluster_addr":"172.21.15.73:6811/56303921","heartbeat_back_addr":"172.21.15.73:6815/56303921","heartbeat_front_addr":"172.21.15.73:6813/56303921","state":["exists","up"]},{"osd":2,"uuid":"4cf96554-e6a0-483a-aa70-66ac8a55f595","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":8,"up_thru":27,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6800","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6801","nonce":544452323}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6802","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6803","nonce":544452323}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6806","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6807","nonce":544452323}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.73:6804","nonce":544452323},{"type":"v1","addr":"172.21.15.73:6805","nonce":544452323}]},"public_addr":"172.21.15.73:6801/544452323","cluster_addr":"172.21.15.73:6803/544452323","heartbeat_back_addr":"172.21.15.73:6807/544452323","heartbeat_front_addr":"172.21.15.73:6805/544452323","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.267945+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.205643+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540701547738038271,"old_weight":0,"last_purged_snaps_scrub":"2024-09-11T08:38:37.060900+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_upmap_primaries":[],"pg_temp":[],"primary_temp":[],"blocklist":{},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-09-11T09:16:05.127 INFO:tasks.ceph:Scrubbing cluster1.osd.0 2024-09-11T09:16:05.127 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.0 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:16:05.287 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-09-11T09:16:05.287 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:16:05.287 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-09-11T09:16:05.303 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 0 2024-09-11T09:16:05.590 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 0 to deep-scrub 2024-09-11T09:16:05.609 INFO:tasks.ceph:Scrubbing cluster1.osd.1 2024-09-11T09:16:05.609 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.1 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:16:05.759 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-09-11T09:16:05.759 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:16:05.759 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-09-11T09:16:05.776 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 1 2024-09-11T09:16:06.064 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 1 to deep-scrub 2024-09-11T09:16:06.083 INFO:tasks.ceph:Scrubbing cluster1.osd.2 2024-09-11T09:16:06.083 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 tell osd.2 config set osd_debug_deep_scrub_sleep 0 2024-09-11T09:16:06.228 INFO:teuthology.orchestra.run.smithi073.stdout:{ 2024-09-11T09:16:06.228 INFO:teuthology.orchestra.run.smithi073.stdout: "success": "osd_debug_deep_scrub_sleep = '' (not observed, change may require restart) " 2024-09-11T09:16:06.228 INFO:teuthology.orchestra.run.smithi073.stdout:} 2024-09-11T09:16:06.244 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 osd deep-scrub 2 2024-09-11T09:16:06.529 INFO:teuthology.orchestra.run.smithi073.stderr:instructed osd(s) 2 to deep-scrub 2024-09-11T09:16:06.548 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T09:16:06.822 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T09:16:06.823 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-09-11T09:16:06.838 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":1591,"stamp":"2024-09-11T09:16:06.240701+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":101,"ondisk_log_size":101,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":300240,"kb_used_data":12496,"kb_used_omap":4,"kb_used_meta":287675,"kb_avail":282815280,"statfs":{"total":289910292480,"available":289602846720,"internally_reserved":0,"allocated":12795904,"data_stored":7384639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":294579535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002902"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559585+0000","last_change":"2024-09-11T08:38:47.626867+0000","last_active":"2024-09-11T09:15:11.559585+0000","last_peered":"2024-09-11T09:15:11.559585+0000","last_clean":"2024-09-11T09:15:11.559585+0000","last_became_active":"2024-09-11T08:38:45.609230+0000","last_became_peered":"2024-09-11T08:38:45.609230+0000","last_unstale":"2024-09-11T09:15:11.559585+0000","last_undegraded":"2024-09-11T09:15:11.559585+0000","last_fullsized":"2024-09-11T09:15:11.559585+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T10:39:06.699918+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021686650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559681+0000","last_change":"2024-09-11T08:38:47.626289+0000","last_active":"2024-09-11T09:15:11.559681+0000","last_peered":"2024-09-11T09:15:11.559681+0000","last_clean":"2024-09-11T09:15:11.559681+0000","last_became_active":"2024-09-11T08:38:45.609290+0000","last_became_peered":"2024-09-11T08:38:45.609290+0000","last_unstale":"2024-09-11T09:15:11.559681+0000","last_undegraded":"2024-09-11T09:15:11.559681+0000","last_fullsized":"2024-09-11T09:15:11.559681+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:31:54.913852+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605272,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559870+0000","last_change":"2024-09-11T08:38:47.625598+0000","last_active":"2024-09-11T09:15:11.559870+0000","last_peered":"2024-09-11T09:15:11.559870+0000","last_clean":"2024-09-11T09:15:11.559870+0000","last_became_active":"2024-09-11T08:38:45.606786+0000","last_became_peered":"2024-09-11T08:38:45.606786+0000","last_unstale":"2024-09-11T09:15:11.559870+0000","last_undegraded":"2024-09-11T09:15:11.559870+0000","last_fullsized":"2024-09-11T09:15:11.559870+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:01:02.286892+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098543599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559911+0000","last_change":"2024-09-11T08:38:47.625546+0000","last_active":"2024-09-11T09:15:11.559911+0000","last_peered":"2024-09-11T09:15:11.559911+0000","last_clean":"2024-09-11T09:15:11.559911+0000","last_became_active":"2024-09-11T08:38:45.606856+0000","last_became_peered":"2024-09-11T08:38:45.606856+0000","last_unstale":"2024-09-11T09:15:11.559911+0000","last_undegraded":"2024-09-11T09:15:11.559911+0000","last_fullsized":"2024-09-11T09:15:11.559911+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T16:04:07.641669+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098068000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"470'3","reported_seq":958,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:05.712091+0000","last_change":"2024-09-11T09:16:05.712133+0000","last_active":"2024-09-11T09:16:05.712091+0000","last_peered":"2024-09-11T09:16:05.712091+0000","last_clean":"2024-09-11T09:16:05.712091+0000","last_became_active":"2024-09-11T08:38:45.603681+0000","last_became_peered":"2024-09-11T08:38:45.603681+0000","last_unstale":"2024-09-11T09:16:05.712091+0000","last_undegraded":"2024-09-11T09:16:05.712091+0000","last_fullsized":"2024-09-11T09:16:05.712091+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'3","last_scrub_stamp":"2024-09-11T09:16:05.711981+0000","last_deep_scrub":"470'3","last_deep_scrub_stamp":"2024-09-11T09:16:05.711981+0000","last_clean_scrub_stamp":"2024-09-11T09:16:05.711981+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:13:53.172084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0021015780000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":944,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.456071+0000","last_change":"2024-09-11T08:38:47.615033+0000","last_active":"2024-09-11T09:15:10.456071+0000","last_peered":"2024-09-11T09:15:10.456071+0000","last_clean":"2024-09-11T09:15:10.456071+0000","last_became_active":"2024-09-11T08:38:45.604617+0000","last_became_peered":"2024-09-11T08:38:45.604617+0000","last_unstale":"2024-09-11T09:15:10.456071+0000","last_undegraded":"2024-09-11T09:15:10.456071+0000","last_fullsized":"2024-09-11T09:15:10.456071+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T17:03:30.831517+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010008059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":945,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:10.447746+0000","last_change":"2024-09-11T08:38:47.615034+0000","last_active":"2024-09-11T09:15:10.447746+0000","last_peered":"2024-09-11T09:15:10.447746+0000","last_clean":"2024-09-11T09:15:10.447746+0000","last_became_active":"2024-09-11T08:38:45.600541+0000","last_became_peered":"2024-09-11T08:38:45.600541+0000","last_unstale":"2024-09-11T09:15:10.447746+0000","last_undegraded":"2024-09-11T09:15:10.447746+0000","last_fullsized":"2024-09-11T09:15:10.447746+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:03:46.935084+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"13'1","reported_seq":946,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.559181+0000","last_change":"2024-09-11T08:38:47.626283+0000","last_active":"2024-09-11T09:15:11.559181+0000","last_peered":"2024-09-11T09:15:11.559181+0000","last_clean":"2024-09-11T09:15:11.559181+0000","last_became_active":"2024-09-11T08:38:45.605202+0000","last_became_peered":"2024-09-11T08:38:45.605202+0000","last_unstale":"2024-09-11T09:15:11.559181+0000","last_undegraded":"2024-09-11T09:15:11.559181+0000","last_fullsized":"2024-09-11T09:15:11.559181+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:44.578859+0000","last_clean_scrub_stamp":"2024-09-11T08:38:44.578859+0000","objects_scrubbed":0,"log_size":1,"log_dups_size":0,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:51:20.908060+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016448249999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"11'97","reported_seq":1074,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:15:11.561308+0000","last_change":"2024-09-11T08:38:43.315778+0000","last_active":"2024-09-11T09:15:11.561308+0000","last_peered":"2024-09-11T09:15:11.561308+0000","last_clean":"2024-09-11T09:15:11.561308+0000","last_became_active":"2024-09-11T08:38:43.314266+0000","last_became_peered":"2024-09-11T08:38:43.314266+0000","last_unstale":"2024-09-11T09:15:11.561308+0000","last_undegraded":"2024-09-11T09:15:11.561308+0000","last_fullsized":"2024-09-11T09:15:11.561308+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T08:38:41.492511+0000","last_clean_scrub_stamp":"2024-09-11T08:38:41.492511+0000","objects_scrubbed":0,"log_size":97,"log_dups_size":0,"ondisk_log_size":97,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T20:05:19.438131+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":4,"ondisk_log_size":4,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":97,"ondisk_log_size":97,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738822,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89044,"kb_used_data":3776,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282796,"statfs":{"total":96636764160,"available":96545583104,"internally_reserved":0,"allocated":3866624,"data_stored":2067955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738821,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89628,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282212,"statfs":{"total":96636764160,"available":96544985088,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738822,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121568,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250272,"statfs":{"total":96636764160,"available":96512278528,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:16:06.839 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2024-09-11T08:38:44.578859+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.839 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2024-09-11T08:38:44.578859+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.839 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2024-09-11T08:38:44.578859+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.840 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2024-09-11T08:38:44.578859+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.840 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2024-09-11T08:38:44.578859+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.840 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2024-09-11T08:38:44.578859+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.840 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2024-09-11T08:38:44.578859+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=44, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.840 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2024-09-11T08:38:41.492511+0000 time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=8, tm_min=38, tm_sec=41, tm_wday=2, tm_yday=255, tm_isdst=-1) <= time.struct_time(tm_year=2024, tm_mon=9, tm_mday=11, tm_hour=9, tm_min=16, tm_sec=4, tm_wday=2, tm_yday=255, tm_isdst=0) 2024-09-11T09:16:06.840 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2024-09-11T09:16:26.841 DEBUG:teuthology.orchestra.run.smithi073:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster cluster1 pg dump --format=json 2024-09-11T09:16:27.147 INFO:teuthology.orchestra.run.smithi073.stdout: 2024-09-11T09:16:27.147 INFO:teuthology.orchestra.run.smithi073.stderr:dumped all 2024-09-11T09:16:27.165 INFO:teuthology.orchestra.run.smithi073.stdout:{"pg_ready":true,"pg_map":{"version":1601,"stamp":"2024-09-11T09:16:26.245498+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590387,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":139,"num_write_kb":1896,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":104,"ondisk_log_size":104,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":3,"num_per_pool_osds":3,"num_per_pool_omap_osds":3,"kb":283115520,"kb_used":300224,"kb_used_data":12496,"kb_used_omap":4,"kb_used_meta":287675,"kb_avail":282815296,"statfs":{"total":289910292480,"available":289602863104,"internally_reserved":0,"allocated":12795904,"data_stored":7384639,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4785,"internal_metadata":294579535},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002970"},"pg_stats":[{"pgid":"2.7","version":"0'0","reported_seq":954,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:11.051646+0000","last_change":"2024-09-11T09:16:11.051724+0000","last_active":"2024-09-11T09:16:11.051646+0000","last_peered":"2024-09-11T09:16:11.051646+0000","last_clean":"2024-09-11T09:16:11.051646+0000","last_became_active":"2024-09-11T08:38:45.609230+0000","last_became_peered":"2024-09-11T08:38:45.609230+0000","last_unstale":"2024-09-11T09:16:11.051646+0000","last_undegraded":"2024-09-11T09:16:11.051646+0000","last_fullsized":"2024-09-11T09:16:11.051646+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:16:11.051445+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:16:11.051445+0000","last_clean_scrub_stamp":"2024-09-11T09:16:11.051445+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:33:51.592285+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0021686650000000002,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":954,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:12.039908+0000","last_change":"2024-09-11T09:16:12.039959+0000","last_active":"2024-09-11T09:16:12.039908+0000","last_peered":"2024-09-11T09:16:12.039908+0000","last_clean":"2024-09-11T09:16:12.039908+0000","last_became_active":"2024-09-11T08:38:45.609290+0000","last_became_peered":"2024-09-11T08:38:45.609290+0000","last_unstale":"2024-09-11T09:16:12.039908+0000","last_undegraded":"2024-09-11T09:16:12.039908+0000","last_fullsized":"2024-09-11T09:16:12.039908+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:16:12.039701+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:16:12.039701+0000","last_clean_scrub_stamp":"2024-09-11T09:16:12.039701+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:34:31.873813+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.001605272,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":955,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:10.041587+0000","last_change":"2024-09-11T09:16:10.041587+0000","last_active":"2024-09-11T09:16:10.041587+0000","last_peered":"2024-09-11T09:16:10.041587+0000","last_clean":"2024-09-11T09:16:10.041587+0000","last_became_active":"2024-09-11T08:38:45.606786+0000","last_became_peered":"2024-09-11T08:38:45.606786+0000","last_unstale":"2024-09-11T09:16:10.041587+0000","last_undegraded":"2024-09-11T09:16:10.041587+0000","last_fullsized":"2024-09-11T09:16:10.041587+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:16:10.041321+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:16:10.041321+0000","last_clean_scrub_stamp":"2024-09-11T09:16:10.041321+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T09:47:39.768670+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098543599999999991,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":955,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:09.082579+0000","last_change":"2024-09-11T09:16:09.082579+0000","last_active":"2024-09-11T09:16:09.082579+0000","last_peered":"2024-09-11T09:16:09.082579+0000","last_clean":"2024-09-11T09:16:09.082579+0000","last_became_active":"2024-09-11T08:38:45.606856+0000","last_became_peered":"2024-09-11T08:38:45.606856+0000","last_unstale":"2024-09-11T09:16:09.082579+0000","last_undegraded":"2024-09-11T09:16:09.082579+0000","last_fullsized":"2024-09-11T09:16:09.082579+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:16:09.082170+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:16:09.082170+0000","last_clean_scrub_stamp":"2024-09-11T09:16:09.082170+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T14:28:04.491066+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.00098068000000000005,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"470'3","reported_seq":959,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:05.712133+0000","last_change":"2024-09-11T09:16:05.712133+0000","last_active":"2024-09-11T09:16:05.712133+0000","last_peered":"2024-09-11T09:16:05.712133+0000","last_clean":"2024-09-11T09:16:05.712133+0000","last_became_active":"2024-09-11T08:38:45.603681+0000","last_became_peered":"2024-09-11T08:38:45.603681+0000","last_unstale":"2024-09-11T09:16:05.712133+0000","last_undegraded":"2024-09-11T09:16:05.712133+0000","last_fullsized":"2024-09-11T09:16:05.712133+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'3","last_scrub_stamp":"2024-09-11T09:16:05.711981+0000","last_deep_scrub":"470'3","last_deep_scrub_stamp":"2024-09-11T09:16:05.711981+0000","last_clean_scrub_stamp":"2024-09-11T09:16:05.711981+0000","objects_scrubbed":1,"log_size":3,"log_dups_size":0,"ondisk_log_size":3,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T21:13:53.172084+0000","scrub_duration":7,"objects_trimmed":0,"snaptrim_duration":0.0021015780000000002,"stat_sum":{"num_bytes":19,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":954,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:08.167625+0000","last_change":"2024-09-11T09:16:08.167625+0000","last_active":"2024-09-11T09:16:08.167625+0000","last_peered":"2024-09-11T09:16:08.167625+0000","last_clean":"2024-09-11T09:16:08.167625+0000","last_became_active":"2024-09-11T08:38:45.604617+0000","last_became_peered":"2024-09-11T08:38:45.604617+0000","last_unstale":"2024-09-11T09:16:08.167625+0000","last_undegraded":"2024-09-11T09:16:08.167625+0000","last_fullsized":"2024-09-11T09:16:08.167625+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:16:08.167377+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:16:08.167377+0000","last_clean_scrub_stamp":"2024-09-11T09:16:08.167377+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:02:15.214425+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010008059999999999,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":954,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:07.199103+0000","last_change":"2024-09-11T09:16:07.199154+0000","last_active":"2024-09-11T09:16:07.199103+0000","last_peered":"2024-09-11T09:16:07.199103+0000","last_clean":"2024-09-11T09:16:07.199103+0000","last_became_active":"2024-09-11T08:38:45.600541+0000","last_became_peered":"2024-09-11T08:38:45.600541+0000","last_unstale":"2024-09-11T09:16:07.199103+0000","last_undegraded":"2024-09-11T09:16:07.199103+0000","last_fullsized":"2024-09-11T09:16:07.199103+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-09-11T09:16:07.198903+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-09-11T09:16:07.198903+0000","last_clean_scrub_stamp":"2024-09-11T09:16:07.198903+0000","objects_scrubbed":0,"log_size":0,"log_dups_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T13:13:20.943793+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0010015600000000001,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,1],"acting":[2,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.3","version":"470'2","reported_seq":957,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:08.039969+0000","last_change":"2024-09-11T09:16:08.039969+0000","last_active":"2024-09-11T09:16:08.039969+0000","last_peered":"2024-09-11T09:16:08.039969+0000","last_clean":"2024-09-11T09:16:08.039969+0000","last_became_active":"2024-09-11T08:38:45.605202+0000","last_became_peered":"2024-09-11T08:38:45.605202+0000","last_unstale":"2024-09-11T09:16:08.039969+0000","last_undegraded":"2024-09-11T09:16:08.039969+0000","last_fullsized":"2024-09-11T09:16:08.039969+0000","mapping_epoch":12,"log_start":"0'0","ondisk_log_start":"0'0","created":12,"last_epoch_clean":13,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'2","last_scrub_stamp":"2024-09-11T09:16:08.039584+0000","last_deep_scrub":"470'2","last_deep_scrub_stamp":"2024-09-11T09:16:08.039584+0000","last_clean_scrub_stamp":"2024-09-11T09:16:08.039584+0000","objects_scrubbed":1,"log_size":2,"log_dups_size":0,"ondisk_log_size":2,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T19:15:09.941532+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0.0016448249999999999,"stat_sum":{"num_bytes":0,"num_objects":1,"num_object_clones":0,"num_object_copies":2,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":1,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"1.0","version":"470'99","reported_seq":1085,"reported_epoch":470,"state":"active+clean","last_fresh":"2024-09-11T09:16:07.046480+0000","last_change":"2024-09-11T09:16:07.046653+0000","last_active":"2024-09-11T09:16:07.046480+0000","last_peered":"2024-09-11T09:16:07.046480+0000","last_clean":"2024-09-11T09:16:07.046480+0000","last_became_active":"2024-09-11T08:38:43.314266+0000","last_became_peered":"2024-09-11T08:38:43.314266+0000","last_unstale":"2024-09-11T09:16:07.046480+0000","last_undegraded":"2024-09-11T09:16:07.046480+0000","last_fullsized":"2024-09-11T09:16:07.046480+0000","mapping_epoch":9,"log_start":"0'0","ondisk_log_start":"0'0","created":9,"last_epoch_clean":10,"parent":"0.0","parent_split_bits":0,"last_scrub":"470'99","last_scrub_stamp":"2024-09-11T09:16:07.046270+0000","last_deep_scrub":"470'99","last_deep_scrub_stamp":"2024-09-11T09:16:07.046270+0000","last_clean_scrub_stamp":"2024-09-11T09:16:07.046270+0000","objects_scrubbed":2,"log_size":99,"log_dups_size":0,"ondisk_log_size":99,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":1,"scrub_schedule":"periodic scrub scheduled @ 2024-09-12T12:48:15.672870+0000","scrub_duration":3,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":19,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":2,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":8192,"data_stored":38,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":5,"ondisk_log_size":5,"up":16,"acting":16,"num_store_stats":3},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":106,"num_read_kb":91,"num_write":137,"num_write_kb":1894,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":1187840,"data_stored":1180736,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":99,"ondisk_log_size":99,"up":2,"acting":2,"num_store_stats":2}],"osd_stats":[{"osd":2,"up_from":8,"seq":34359738826,"num_pgs":3,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89040,"kb_used_data":3776,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282800,"statfs":{"total":96636764160,"available":96545587200,"internally_reserved":0,"allocated":3866624,"data_stored":2067955,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,1],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":1,"up_from":8,"seq":34359738825,"num_pgs":9,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":89624,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":85246,"kb_avail":94282216,"statfs":{"total":96636764160,"available":96544989184,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":87292357},"hb_peers":[0,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]},{"osd":0,"up_from":8,"seq":34359738826,"num_pgs":6,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":94371840,"kb_used":121560,"kb_used_data":4360,"kb_used_omap":1,"kb_used_meta":117182,"kb_avail":94250280,"statfs":{"total":96636764160,"available":96512286720,"internally_reserved":0,"allocated":4464640,"data_stored":2658342,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1595,"internal_metadata":119994821},"hb_peers":[1,2],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":4096,"data_stored":19,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":2,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-09-11T09:16:27.166 DEBUG:teuthology.orchestra.run.smithi073:> sudo ceph --cluster cluster1 config set global mon_health_to_clog false 2024-09-11T09:16:27.492 INFO:teuthology.misc:Shutting down mds daemons... 2024-09-11T09:16:27.492 INFO:teuthology.misc:Shutting down osd daemons... 2024-09-11T09:16:27.492 DEBUG:tasks.ceph.cluster1.osd.0:waiting for process to exit 2024-09-11T09:16:27.492 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:16:30.330 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 is failed for ~0s 2024-09-11T09:16:33.594 INFO:tasks.ceph.cluster1.osd.0:Stopped 2024-09-11T09:16:33.595 DEBUG:tasks.ceph.cluster1.osd.1:waiting for process to exit 2024-09-11T09:16:33.595 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:16:33.669 INFO:tasks.ceph.cluster1.osd.1:Stopped 2024-09-11T09:16:33.669 DEBUG:tasks.ceph.cluster1.osd.2:waiting for process to exit 2024-09-11T09:16:33.670 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:16:35.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 is failed for ~0s 2024-09-11T09:16:35.533 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.0 has been restored 2024-09-11T09:16:39.772 INFO:tasks.ceph.cluster1.osd.2:Stopped 2024-09-11T09:16:39.772 INFO:teuthology.misc:Shutting down mgr daemons... 2024-09-11T09:16:39.772 DEBUG:tasks.ceph.cluster1.mgr.x:waiting for process to exit 2024-09-11T09:16:39.772 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:16:39.802 INFO:tasks.ceph.cluster1.mgr.x:Stopped 2024-09-11T09:16:39.802 INFO:teuthology.misc:Shutting down mon daemons... 2024-09-11T09:16:39.803 DEBUG:tasks.ceph.cluster1.mon.a:waiting for process to exit 2024-09-11T09:16:39.803 INFO:teuthology.orchestra.run:waiting for 300 2024-09-11T09:16:39.861 INFO:tasks.ceph.cluster1.mon.a:Stopped 2024-09-11T09:16:39.862 INFO:tasks.ceph:Checking cluster log for badness... 2024-09-11T09:16:39.862 DEBUG:teuthology.orchestra.run.smithi073:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/cluster1.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(OSD_SLOW_PING_TIME' | head -n 1 2024-09-11T09:16:39.916 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-0 on ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T09:16:39.916 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-0 2024-09-11T09:16:40.213 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-1 on ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T09:16:40.214 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-1 2024-09-11T09:16:40.437 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/cluster1-2 on ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T09:16:40.437 DEBUG:teuthology.orchestra.run.smithi073:> sync && sudo umount -f /var/lib/ceph/osd/cluster1-2 2024-09-11T09:16:40.534 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon cluster1.osd.2 has been restored 2024-09-11T09:16:40.629 INFO:tasks.ceph:Archiving mon data... 2024-09-11T09:16:40.629 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/mon/cluster1-a to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/data/cluster1.mon.a.tgz 2024-09-11T09:16:40.629 DEBUG:teuthology.orchestra.run.smithi073:> mktemp 2024-09-11T09:16:40.635 INFO:teuthology.orchestra.run.smithi073.stdout:/tmp/tmp.QKhLNC3rgy 2024-09-11T09:16:40.636 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar cz -f - -C /var/lib/ceph/mon/cluster1-a -- . > /tmp/tmp.QKhLNC3rgy 2024-09-11T09:16:40.785 DEBUG:teuthology.orchestra.run.smithi073:> sudo chmod 0666 /tmp/tmp.QKhLNC3rgy 2024-09-11T09:16:40.847 DEBUG:teuthology.orchestra.remote:smithi073:/tmp/tmp.QKhLNC3rgy is 408KB 2024-09-11T09:16:40.909 DEBUG:teuthology.orchestra.run.smithi073:> rm -fr /tmp/tmp.QKhLNC3rgy 2024-09-11T09:16:40.914 INFO:tasks.ceph:Cleaning ceph cluster... 2024-09-11T09:16:40.914 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T09:16:40.956 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -rf -- /etc/ceph/cluster1.conf /etc/ceph/cluster1.keyring /home/ubuntu/cephtest/cluster1.data /home/ubuntu/cephtest/cluster1.monmap /home/ubuntu/cephtest/../*.pid 2024-09-11T09:16:41.015 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T09:16:41.021 INFO:teuthology.util.scanner:summary_data or yaml_file is empty! 2024-09-11T09:16:41.021 INFO:tasks.ceph:Archiving crash dumps... 2024-09-11T09:16:41.023 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/remote/smithi073/crash 2024-09-11T09:16:41.024 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T09:16:41.072 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/lib/ceph/crash to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/remote/smithi077/crash 2024-09-11T09:16:41.073 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/lib/ceph/crash -- . 2024-09-11T09:16:41.086 INFO:tasks.ceph:Compressing logs... 2024-09-11T09:16:41.086 DEBUG:teuthology.orchestra.run.smithi073:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T09:16:41.118 DEBUG:teuthology.orchestra.run.smithi077:> time sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-09-11T09:16:41.137 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29985.log 2024-09-11T09:16:41.138 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.audit.log 2024-09-11T09:16:41.138 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.29985.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32305.log 2024-09-11T09:16:41.138 INFO:teuthology.orchestra.run.smithi073.stderr: 80.2% -- replaced with /var/log/ceph/cluster1-client.admin.29985.log.gz 2024-09-11T09:16:41.139 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1.audit.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32229.log 2024-09-11T09:16:41.140 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32305.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32073.log 2024-09-11T09:16:41.140 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32305.log.gz 2024-09-11T09:16:41.140 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32229.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30005.log 2024-09-11T09:16:41.140 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32229.log.gz 2024-09-11T09:16:41.143 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32073.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29945.log 2024-09-11T09:16:41.143 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32073.log.gz 2024-09-11T09:16:41.143 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30005.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30005.log.gz 2024-09-11T09:16:41.143 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-mon.a.log 2024-09-11T09:16:41.143 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.29945.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25437.log 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29945.log.gz 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31995.log 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr: 92.9% -- replaced with /var/log/ceph/cluster1.audit.log.gz 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.25437.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25332.log 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25437.log.gz 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.31995.log: gzip -5 --verbose 0.0% -- -- replaced with /var/log/ceph/cluster1-client.admin.31995.log.gz 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr: /var/log/ceph/cluster1-client.admin.32047.log 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.25332.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32099.log 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25332.log.gz 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32047.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32047.log.gz 2024-09-11T09:16:41.144 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30242.log 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32099.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30169.log 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32099.log.gz 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30242.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30242.log.gz 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30161.log 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose --/var/log/ceph/cluster1-client.admin.30169.log: /var/log/ceph/cluster1-client.admin.30090.log 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30169.log.gz 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30161.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30161.log.gz 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.29900.log 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30090.log: gzip -5 --verbose -- /var/log/ceph/cluster2-mgr.x.log 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30090.log.gz 2024-09-11T09:16:41.149 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.29900.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.29900.log.gz 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30317.log 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32203.log 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster2-mgr.x.log: /var/log/ceph/cluster1-client.admin.30317.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30317.log.gz 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25284.log 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32203.log: gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21535.log 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32203.log.gz 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.25284.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25284.log.gz 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32177.log 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/ceph.tmp-client.admin.21535.log: gzip -5 --verbose -- /var/log/ceph/cluster1-mgr.x.log 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21535.log.gz 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32177.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32177.log.gz 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.2.log 2024-09-11T09:16:41.150 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-mgr.x.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30091.log 2024-09-11T09:16:41.154 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-osd.2.log: gzip -5 --verbose -- /var/log/ceph/cluster1-osd.0.log 2024-09-11T09:16:41.154 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30091.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30343.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30091.log.gz 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30243.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30343.log: 0.0%gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32125.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.30343.log.gz 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30243.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25367.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0%/var/log/ceph/cluster1-client.admin.32125.log: -- replaced with /var/log/ceph/cluster1-client.admin.30243.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25308.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32125.log.gz 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.25367.log: gzip 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25367.log.gz -5 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: --verbose -- /var/log/ceph/cluster1-client.admin.32281.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.25308.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.28055.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25308.log.gz 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32281.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30088.log 2024-09-11T09:16:41.155 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32281.log.gz 2024-09-11T09:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.28055.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.28055.log.gz 2024-09-11T09:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30369.log 2024-09-11T09:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30088.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.25190.log 2024-09-11T09:16:41.156 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30088.log.gz 2024-09-11T09:16:41.163 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30369.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30369.log.gz 2024-09-11T09:16:41.163 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1.log 2024-09-11T09:16:41.163 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.25190.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32255.log 2024-09-11T09:16:41.164 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.25190.log.gz 2024-09-11T09:16:41.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30247.log 2024-09-11T09:16:41.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30168.log 2024-09-11T09:16:41.164 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32255.log.gz 2024-09-11T09:16:41.164 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30247.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30247.log.gz 2024-09-11T09:16:41.165 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30057.log 2024-09-11T09:16:41.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30168.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30031.log 2024-09-11T09:16:41.165 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30168.log.gz 2024-09-11T09:16:41.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30057.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30057.log.gz 2024-09-11T09:16:41.165 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32021.log 2024-09-11T09:16:41.165 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.30031.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32151.log 2024-09-11T09:16:41.166 INFO:teuthology.orchestra.run.smithi073.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30031.log.gz 2024-09-11T09:16:41.166 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32021.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32021.log.gz 2024-09-11T09:16:41.166 INFO:teuthology.orchestra.run.smithi073.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-osd.1.log 2024-09-11T09:16:41.166 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-client.admin.32151.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.32151.log.gz 2024-09-11T09:16:41.166 INFO:teuthology.orchestra.run.smithi073.stderr: 88.0% -- replaced with /var/log/ceph/cluster1.log.gz 2024-09-11T09:16:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57537.log 2024-09-11T09:16:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39897.log 2024-09-11T09:16:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57537.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49532.log 2024-09-11T09:16:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39897.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61072.log 2024-09-11T09:16:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49532.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39897.log.gz 2024-09-11T09:16:41.204 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45208.log 2024-09-11T09:16:41.205 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61072.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42496.log 2024-09-11T09:16:41.205 INFO:teuthology.orchestra.run.smithi077.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61072.log.gz 2024-09-11T09:16:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45208.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54235.log 2024-09-11T09:16:41.206 INFO:teuthology.orchestra.run.smithi077.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57537.log.gz 2024-09-11T09:16:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58199.log 2024-09-11T09:16:41.206 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42496.log.gz 2024-09-11T09:16:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54235.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49532.log.gz 2024-09-11T09:16:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39668.log 2024-09-11T09:16:41.206 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58199.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55073.log 2024-09-11T09:16:41.207 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45208.log.gz 2024-09-11T09:16:41.207 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54235.log.gz 2024-09-11T09:16:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39668.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33312.log 2024-09-11T09:16:41.207 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58199.log.gz 2024-09-11T09:16:41.207 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39668.log.gz 2024-09-11T09:16:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53307.log 2024-09-11T09:16:41.207 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55073.log: /var/log/ceph/cluster2-client.mirror.33312.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43328.log 2024-09-11T09:16:41.208 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55073.log.gz 2024-09-11T09:16:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53307.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37621.log 2024-09-11T09:16:41.208 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33312.log.gz 2024-09-11T09:16:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57723.log 2024-09-11T09:16:41.208 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53307.log.gz 2024-09-11T09:16:41.208 INFO:teuthology.orchestra.run.smithi077.stderr: 79.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43328.log.gz 2024-09-11T09:16:41.208 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5/var/log/ceph/cluster1-client.mirror.37621.log: --verbose -- /var/log/ceph/cluster1-client.mirror.55049.log 2024-09-11T09:16:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57723.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48245.log 2024-09-11T09:16:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/cluster1-client.mirror.55049.log: -5 --verbose -- /var/log/ceph/cluster2-client.admin.30034.log 2024-09-11T09:16:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48245.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55049.log.gz 2024-09-11T09:16:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48569.log 2024-09-11T09:16:41.209 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48245.log.gz 2024-09-11T09:16:41.209 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65452.log 2024-09-11T09:16:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30034.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30034.log.gz 2024-09-11T09:16:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48569.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59724.log 2024-09-11T09:16:41.210 INFO:teuthology.orchestra.run.smithi077.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57723.log.gz 2024-09-11T09:16:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65452.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47305.log 2024-09-11T09:16:41.210 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65452.log.gz 2024-09-11T09:16:41.210 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59724.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39086.log 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.48569.log.gz 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59724.log.gz 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51060.log 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47305.log: /var/log/ceph/cluster2-client.mirror.39086.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65499.log 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47305.log.gz 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51060.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52954.log 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51060.log.gz 2024-09-11T09:16:41.211 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39086.log.gz 2024-09-11T09:16:41.212 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65499.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50433.log 2024-09-11T09:16:41.212 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37621.log.gz 2024-09-11T09:16:41.212 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57145.log 2024-09-11T09:16:41.212 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52954.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65499.log.gz 2024-09-11T09:16:41.212 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50433.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39690.log 2024-09-11T09:16:41.212 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57145.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50745.log 2024-09-11T09:16:41.212 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.50433.log.gz 2024-09-11T09:16:41.213 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39690.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57145.log.gz 2024-09-11T09:16:41.213 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47112.log 2024-09-11T09:16:41.213 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31187.log 2024-09-11T09:16:41.213 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50745.log: /var/log/ceph/cluster1-client.mirror.47112.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60172.log 2024-09-11T09:16:41.213 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47112.log.gz 2024-09-11T09:16:41.213 INFO:teuthology.orchestra.run.smithi077.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.52954.log.gz 2024-09-11T09:16:41.213 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31187.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52618.log 2024-09-11T09:16:41.214 INFO:teuthology.orchestra.run.smithi077.stderr: 55.3% -- replaced with /var/log/ceph/cluster1-client.mirror.31187.log.gz 2024-09-11T09:16:41.214 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60172.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50745.log.gz 2024-09-11T09:16:41.214 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55352.log 2024-09-11T09:16:41.214 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60172.log.gz 2024-09-11T09:16:41.214 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36419.log 2024-09-11T09:16:41.214 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52618.log: /var/log/ceph/cluster1-client.mirror.55352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66878.log 2024-09-11T09:16:41.214 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose 87.7% -- /var/log/ceph/cluster1-client.mirror.64461.log -- replaced with /var/log/ceph/cluster1-client.mirror.55352.log.gz 2024-09-11T09:16:41.215 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.215 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36419.log: /var/log/ceph/cluster2-client.admin.66878.log: 0.0%gzip -- replaced with /var/log/ceph/cluster2-client.admin.66878.log.gz 2024-09-11T09:16:41.215 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36763.log 2024-09-11T09:16:41.215 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.36419.log.gz 2024-09-11T09:16:41.215 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64461.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45931.log 2024-09-11T09:16:41.215 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64461.log.gz 2024-09-11T09:16:41.216 INFO:teuthology.orchestra.run.smithi077.stderr: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52618.log.gz 2024-09-11T09:16:41.216 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34737.log 2024-09-11T09:16:41.216 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36763.log: 76.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36763.log.gz 2024-09-11T09:16:41.216 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45931.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41812.log 2024-09-11T09:16:41.216 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.34737.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44011.log 2024-09-11T09:16:41.217 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.34737.log.gz 2024-09-11T09:16:41.217 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41812.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45931.log.gz 2024-09-11T09:16:41.217 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.31651.log 2024-09-11T09:16:41.217 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60217.log 2024-09-11T09:16:41.217 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44011.log: /var/log/ceph/cluster2-client.mirror.2.31651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42239.log 2024-09-11T09:16:41.217 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41812.log.gz 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60217.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63769.log 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60217.log.gz 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44011.log.gz 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42239.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32955.log 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63769.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39197.log 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.63769.log.gz 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.32955.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42239.log.gz 2024-09-11T09:16:41.218 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31570.log 2024-09-11T09:16:41.219 INFO:teuthology.orchestra.run.smithi077.stderr: 82.0% -- replaced with /var/log/ceph/cluster1-client.mirror.32955.log.gz 2024-09-11T09:16:41.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39197.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54981.log 2024-09-11T09:16:41.219 INFO:teuthology.orchestra.run.smithi077.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39197.log.gz 2024-09-11T09:16:41.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31570.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37664.log 2024-09-11T09:16:41.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54981.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31570.log.gz 2024-09-11T09:16:41.219 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52304.log 2024-09-11T09:16:41.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37664.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46690.log 2024-09-11T09:16:41.220 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54981.log.gz 2024-09-11T09:16:41.220 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52304.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38543.log 2024-09-11T09:16:41.220 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52304.log.gz 2024-09-11T09:16:41.220 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46690.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44228.log 2024-09-11T09:16:41.220 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30646.log 2024-09-11T09:16:41.220 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38543.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46690.log.gz 2024-09-11T09:16:41.221 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44228.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61230.log 2024-09-11T09:16:41.221 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38543.log.gz 2024-09-11T09:16:41.221 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.30646.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54669.log 2024-09-11T09:16:41.221 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30646.log.gz 2024-09-11T09:16:41.221 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44228.log.gz 2024-09-11T09:16:41.221 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47831.log 2024-09-11T09:16:41.221 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61230.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66074.log 2024-09-11T09:16:41.222 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54669.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61230.log.gz 2024-09-11T09:16:41.222 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47831.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39880.log 2024-09-11T09:16:41.222 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47831.log.gz 2024-09-11T09:16:41.222 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66074.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33901.log 2024-09-11T09:16:41.222 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.54669.log.gz 2024-09-11T09:16:41.222 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39880.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66074.log.gz 2024-09-11T09:16:41.222 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40188.log 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39880.log.gz 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33901.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58938.log 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64135.log 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40188.log: /var/log/ceph/cluster2-client.mirror.58938.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40937.log 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35954.log 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64135.log: /var/log/ceph/cluster2-client.mirror.40937.log: 86.3% -- replaced with /var/log/ceph/cluster1-client.mirror.64135.log.gz 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37664.log.gz 2024-09-11T09:16:41.223 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47329.log 2024-09-11T09:16:41.224 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40937.log.gz 2024-09-11T09:16:41.224 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65164.log 2024-09-11T09:16:41.224 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35954.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58938.log.gz/var/log/ceph/cluster1-client.mirror.47329.log: gzip 2024-09-11T09:16:41.224 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50586.log 2024-09-11T09:16:41.224 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47329.log.gz 2024-09-11T09:16:41.224 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65164.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65164.log.gz 2024-09-11T09:16:41.224 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38669.log 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55373.log 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50586.log: /var/log/ceph/cluster1-client.mirror.38669.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40710.log 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50586.log.gz 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38669.log.gz/var/log/ceph/cluster2-client.mirror.55373.log: 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.38903.log 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60982.log 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.1.38903.log: /var/log/ceph/cluster2-client.mirror.40710.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55373.log.gz 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35954.log.gz 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43841.log 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45143.log 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43841.log: 85.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60982.log.gz 2024-09-11T09:16:41.225 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47759.log 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45143.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43841.log.gz 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45143.log.gz 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37922.log 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47759.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40129.log 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8%/var/log/ceph/cluster1-client.mirror.37922.log: -- replaced with /var/log/ceph/cluster2-client.mirror.47759.log.gz 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66472.log 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40129.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40129.log.gz 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54122.log 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66472.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.1.38903.log.gz 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66472.log.gz 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48965.log 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54122.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38383.log 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48965.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54122.log.gz 2024-09-11T09:16:41.226 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37922.log.gz 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49779.log 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48965.log.gz 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38383.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61320.log 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53648.log 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38383.log.gz 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49779.log: /var/log/ceph/cluster1-client.mirror.61320.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64328.log 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr: 86.6%/var/log/ceph/cluster1-client.mirror.53648.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61320.log.gz 2024-09-11T09:16:41.227 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53648.log.gz 2024-09-11T09:16:41.257 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36303.log 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64328.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63358.log 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49779.log.gz 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.36303.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64328.log.gz 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61837.log 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/cluster1-client.admin.36303.log.gz 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63358.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31392.log 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr: 79.8%/var/log/ceph/cluster1-client.mirror.61837.log: -- replaced with /var/log/ceph/cluster2-client.mirror.63358.log.gz 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46526.log 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9%/var/log/ceph/cluster2-client.admin.31392.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61837.log.gz 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52260.log 2024-09-11T09:16:41.258 INFO:teuthology.orchestra.run.smithi077.stderr: 67.6% -- replaced with /var/log/ceph/cluster2-client.admin.31392.log.gz 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46526.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51810.log 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55140.log 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52260.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46526.log.gz 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51810.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37343.log 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55140.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52260.log.gz 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51810.log.gz 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52525.log 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37343.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59166.log 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52525.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.55140.log.gz 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52525.log.gz 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50883.log 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59166.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54392.log 2024-09-11T09:16:41.259 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59166.log.gz 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50883.log: 92.9% -- replaced with /var/log/ceph/cluster2-client.mirror.2.31651.log.gz 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59812.log 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr: 87.2% -- replaced with /var/log/ceph/cluster1-client.mirror.50883.log.gz 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54392.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54544.log 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59812.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48225.log 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59812.log.gz 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54544.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54544.log.gz 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54392.log.gz 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54050.log 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37211.log 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48225.log: /var/log/ceph/cluster1-client.mirror.54050.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50811.log 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37343.log.gz 2024-09-11T09:16:41.260 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54050.log.gz 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9%/var/log/ceph/cluster2-client.mirror.37211.log: -- replaced with /var/log/ceph/cluster2-client.mirror.48225.log.gz 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.37211.log.gz 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61635.log 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50811.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35327.log 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50811.log.gz 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61635.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35468.log 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61635.log.gz 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33290.log 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35327.log: /var/log/ceph/cluster2-client.mirror.35468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62017.log 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.35327.log.gz 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33290.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58414.log 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50313.log 2024-09-11T09:16:41.261 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.62017.log: /var/log/ceph/cluster2-client.mirror.58414.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.62017.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63912.log 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58414.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50313.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50313.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45047.log 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39552.log 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63912.log: /var/log/ceph/cluster2-client.mirror.45047.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63912.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45047.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30321.log 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31170.log 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39552.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39552.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30321.log: 0.0% 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35468.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52597.log 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.admin.30321.log.gz 2024-09-11T09:16:41.262 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33290.log.gz 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30898.log 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.31170.log: /var/log/ceph/cluster1-client.mirror.52597.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56731.log 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr: 66.2% -- replaced with /var/log/ceph/cluster1-client.admin.31170.log.gz 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.30898.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33247.log 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52597.log.gz 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr: 74.1% -- replaced with /var/log/ceph/cluster1-client.mirror.30898.log.gz 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56731.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42538.log 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50542.log 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33247.log: /var/log/ceph/cluster2-client.mirror.42538.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30391.log 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42538.log.gz/var/log/ceph/cluster2-client.mirror.50542.log: 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50542.log.gz 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56731.log.gz 2024-09-11T09:16:41.263 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47184.log 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30391.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65543.log 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30391.log.gz 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47184.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31289.log 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65102.log 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0%/var/log/ceph/cluster2-client.mirror.65543.log: -- replaced with /var/log/ceph/cluster1-client.mirror.47184.log.gz 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31289.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31289.log.gz 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65543.log.gz 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66595.log 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65102.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.30934.log 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.33247.log.gz 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.66595.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65102.log.gz 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58608.log 2024-09-11T09:16:41.264 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66595.log.gz 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55819.log 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58608.log: /var/log/ceph/cluster2-client.mirror.2.30934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59083.log 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr: 71.3%/var/log/ceph/cluster2-client.mirror.55819.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49511.log 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.2.30934.log.gz 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58608.log.gz 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59083.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59083.log.gz 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55819.log.gz 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33857.log 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49686.log 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49511.log: /var/log/ceph/cluster1-client.mirror.33857.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49300.log 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49511.log.gz 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53894.log 2024-09-11T09:16:41.265 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49686.log: /var/log/ceph/cluster1-client.mirror.49300.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46598.log 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53894.log: 87.7% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49300.log.gz -- replaced with /var/log/ceph/cluster1-client.mirror.49686.log.gz 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25305.log 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63727.log 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46598.log: /var/log/ceph/cluster2-client.admin.25305.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25305.log.gz 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53894.log.gz 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46598.log.gz 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37601.log 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41582.log 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63727.log: /var/log/ceph/cluster2-client.mirror.37601.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38881.log 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41582.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63727.log.gz 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65353.log 2024-09-11T09:16:41.266 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.33857.log.gz 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38881.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58240.log 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65353.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65353.log.gz 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38881.log.gz 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34652.log 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58240.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49821.log 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58240.log.gz 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34652.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53263.log 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49821.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34652.log.gz 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44296.log 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49821.log.gz 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53263.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37601.log.gz 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31136.log 2024-09-11T09:16:41.267 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43424.log 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44296.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53263.log.gz 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.44296.log.gz 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.31136.log: 80.3% -- replaced with /var/log/ceph/cluster1-client.admin.31136.log.gz 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35488.log 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43424.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54589.log 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35488.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48052.log 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5%/var/log/ceph/cluster2-client.mirror.54589.log: -- replaced with /var/log/ceph/cluster2-client.mirror.43424.log.gz 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61950.log 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48052.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48052.log.gz 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66015.log 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54589.log.gz 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61950.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61950.log.gz 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.43567.log 2024-09-11T09:16:41.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66015.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30596.log 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.43567.log: 93.6% -- replaced with /var/log/ceph/cluster1-client.mirror.43567.log.gz 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49903.log 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.30596.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30596.log.gz 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.35488.log.gz 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59352.log 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49903.log: 91.5% -- replaced with /var/log/ceph/cluster2-client.mirror.66015.log.gz 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49903.log.gz 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39389.log 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59352.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30059.log 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39389.log: 88.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35039.log 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.59352.log.gz 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39389.log.gz 2024-09-11T09:16:41.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30059.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56979.log 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr: 80.3% -- replaced with /var/log/ceph/cluster2-client.admin.30059.log.gz 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51673.log 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35039.log: /var/log/ceph/cluster2-client.mirror.56979.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43345.log 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51673.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35039.log.gz 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54827.log 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.51673.log.gz 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43345.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43345.log.gz 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56979.log.gz 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51219.log 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52439.log 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.51219.log: -- replaced with /var/log/ceph/cluster1-client.mirror.54827.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40146.log 2024-09-11T09:16:41.270 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47693.log 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52439.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51219.log.gz/var/log/ceph/cluster2-client.mirror.40146.log: 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52439.log.gz 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40146.log.gz 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40651.log 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47693.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64716.log 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40651.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39917.log 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40651.log.gz 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47693.log.gz 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64716.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64716.log.gz 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.64371.log 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40668.log 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.64371.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39917.log.gz 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37902.log 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44248.log 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37902.log: /var/log/ceph/cluster2-client.mirror.40668.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40668.log.gz 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58856.log 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.64371.log.gz 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44248.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44248.log.gz 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52039.log 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58856.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47879.log 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52039.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58856.log.gz 2024-09-11T09:16:41.282 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52039.log.gz 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46437.log 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49232.log 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37902.log.gz/var/log/ceph/cluster1-client.mirror.47879.log: 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46437.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47879.log.gz 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46437.log.gz 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66322.log 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49232.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57289.log 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.66322.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39214.log 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.66322.log.gz 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57289.log: -- replaced with /var/log/ceph/cluster2-client.mirror.49232.log.gz 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57289.log.gz 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48878.log 2024-09-11T09:16:41.283 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39214.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61590.log 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48878.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38031.log 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% 89.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48878.log.gz 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.39214.log.gz 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61590.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61590.log.gz 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.64135.log 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62116.log 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38031.log: /var/log/ceph/cluster2-client.mirror.1.64135.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45485.log 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.62116.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62116.log.gz 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51305.log 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45485.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45485.log.gz 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66741.log 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51305.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51305.log.gz 2024-09-11T09:16:41.284 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37750.log 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.64135.log.gz 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38031.log.gz 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.66741.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66741.log.gz 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36966.log 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37750.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62099.log 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36966.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64546.log 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36966.log.gz 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.62099.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.62099.log.gz 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43032.log 2024-09-11T09:16:41.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64546.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61545.log 2024-09-11T09:16:41.286 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43032.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43032.log.gz 2024-09-11T09:16:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34567.log 2024-09-11T09:16:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61545.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61545.log.gz 2024-09-11T09:16:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 91.7% --verbose -- -- replaced with /var/log/ceph/cluster2-client.mirror.64546.log.gz /var/log/ceph/cluster2-client.mirror.63748.log 2024-09-11T09:16:41.287 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.287 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34567.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37750.log.gz 2024-09-11T09:16:41.289 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34567.log.gz 2024-09-11T09:16:41.289 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58137.log 2024-09-11T09:16:41.290 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63748.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.66789.log 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58137.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41309.log 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63748.log.gz 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58137.log.gz/var/log/ceph/cluster1-client.admin.66789.log: 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.66789.log.gz 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34450.log 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41309.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57620.log 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34450.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41309.log.gz 2024-09-11T09:16:41.292 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41083.log 2024-09-11T09:16:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43824.log 2024-09-11T09:16:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57620.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34450.log.gz 2024-09-11T09:16:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41083.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58876.log 2024-09-11T09:16:41.294 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57620.log.gz 2024-09-11T09:16:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43824.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43824.log.gz 2024-09-11T09:16:41.294 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33355.log 2024-09-11T09:16:41.294 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41083.log.gz 2024-09-11T09:16:41.295 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58876.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58876.log.gz 2024-09-11T09:16:41.296 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49099.log 2024-09-11T09:16:41.298 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.33355.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54303.log 2024-09-11T09:16:41.298 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49099.log: 87.6%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54074.log 2024-09-11T09:16:41.298 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.admin.33355.log.gz 2024-09-11T09:16:41.298 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.54303.log: -- replaced with /var/log/ceph/cluster1-client.mirror.49099.log.gz 2024-09-11T09:16:41.298 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34037.log 2024-09-11T09:16:41.298 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54303.log.gz 2024-09-11T09:16:41.298 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49144.log 2024-09-11T09:16:41.299 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54074.log: /var/log/ceph/cluster2-client.mirror.34037.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54074.log.gz 2024-09-11T09:16:41.299 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.34037.log.gz 2024-09-11T09:16:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30922.log 2024-09-11T09:16:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49144.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30243.log 2024-09-11T09:16:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.30922.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60330.log 2024-09-11T09:16:41.300 INFO:teuthology.orchestra.run.smithi077.stderr: 80.7% -- replaced with /var/log/ceph/cluster1-client.mirror.30922.log.gz 2024-09-11T09:16:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30243.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30243.log.gz 2024-09-11T09:16:41.300 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56086.log 2024-09-11T09:16:41.302 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50791.log 2024-09-11T09:16:41.302 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49144.log.gz 2024-09-11T09:16:41.302 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60330.log: /var/log/ceph/cluster2-client.mirror.56086.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60330.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34610.log 2024-09-11T09:16:41.302 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.302 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50791.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56200.log 2024-09-11T09:16:41.302 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56086.log.gz 2024-09-11T09:16:41.302 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34610.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50791.log.gz 2024-09-11T09:16:41.303 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41289.log 2024-09-11T09:16:41.303 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56200.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58566.log 2024-09-11T09:16:41.304 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.56200.log.gz 2024-09-11T09:16:41.304 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41289.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54713.log 2024-09-11T09:16:41.304 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58566.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41289.log.gz 2024-09-11T09:16:41.305 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46088.log 2024-09-11T09:16:41.306 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54713.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54713.log.gz 2024-09-11T09:16:41.306 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25374.log 2024-09-11T09:16:41.307 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58566.log.gz 2024-09-11T09:16:41.307 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46088.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52817.log 2024-09-11T09:16:41.308 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5%/var/log/ceph/cluster2-client.admin.25374.log: -- replaced with /var/log/ceph/cluster2-client.mirror.34610.log.gz 2024-09-11T09:16:41.308 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25374.log.gz 2024-09-11T09:16:41.308 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59062.log 2024-09-11T09:16:41.309 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52817.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60487.log 2024-09-11T09:16:41.309 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59062.log: 87.5% 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46088.log.gz 2024-09-11T09:16:41.309 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52817.log.gz 2024-09-11T09:16:41.309 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54279.log 2024-09-11T09:16:41.310 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60487.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54939.log 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60487.log.gz 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54279.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30721.log 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54279.log.gz 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54939.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59062.log.gz 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41733.log 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61387.log 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54939.log.gz/var/log/ceph/cluster2-client.admin.30721.log: 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41733.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30721.log.gz 2024-09-11T09:16:41.312 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53403.log 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61387.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35793.log 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61387.log.gz 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.41733.log.gz 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53403.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49007.log 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53403.log.gz/var/log/ceph/cluster2-client.mirror.35793.log: 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34133.log 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49007.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35793.log.gz 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50562.log 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61027.log 2024-09-11T09:16:41.315 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49007.log.gz 2024-09-11T09:16:41.317 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34133.log: /var/log/ceph/cluster2-client.mirror.50562.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.34133.log.gz 2024-09-11T09:16:41.317 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50562.log.gz 2024-09-11T09:16:41.317 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59579.log 2024-09-11T09:16:41.317 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61027.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.40811.log 2024-09-11T09:16:41.318 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61027.log.gz 2024-09-11T09:16:41.318 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59579.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50496.log 2024-09-11T09:16:41.318 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47464.log 2024-09-11T09:16:41.318 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.2.40811.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59579.log.gz 2024-09-11T09:16:41.319 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50496.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50496.log.gz 2024-09-11T09:16:41.319 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34764.log 2024-09-11T09:16:41.319 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41481.log 2024-09-11T09:16:41.320 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47464.log: /var/log/ceph/cluster1-client.admin.34764.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.34764.log.gz 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47464.log.gz 2024-09-11T09:16:41.320 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.320 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51989.log 2024-09-11T09:16:41.320 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47606.log 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41481.log: /var/log/ceph/cluster2-client.mirror.51989.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41481.log.gz 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63375.log 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8%/var/log/ceph/cluster1-client.mirror.47606.log: -- replaced with /var/log/ceph/cluster2-client.mirror.51989.log.gz 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47606.log.gz 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44918.log 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30131.log 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63375.log: /var/log/ceph/cluster1-client.mirror.44918.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44918.log.gz 2024-09-11T09:16:41.327 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63375.log.gz 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61297.log 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45533.log 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30131.log: /var/log/ceph/cluster1-client.mirror.61297.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30131.log.gz 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61297.log.gz 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53624.log 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54002.log 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45533.log: /var/log/ceph/cluster1-client.mirror.53624.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45533.log.gz 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53624.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54098.log 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54002.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54002.log.gz 2024-09-11T09:16:41.328 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47630.log 2024-09-11T09:16:41.329 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59000.log 2024-09-11T09:16:41.330 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54098.log: /var/log/ceph/cluster1-client.mirror.47630.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54098.log.gz 2024-09-11T09:16:41.330 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57806.log 2024-09-11T09:16:41.330 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47630.log.gz 2024-09-11T09:16:41.330 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59000.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59000.log.gz 2024-09-11T09:16:41.331 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38403.log 2024-09-11T09:16:41.331 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44728.log 2024-09-11T09:16:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57806.log: /var/log/ceph/cluster2-client.mirror.38403.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57806.log.gz 2024-09-11T09:16:41.332 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53103.log 2024-09-11T09:16:41.332 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38403.log.gz 2024-09-11T09:16:41.333 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44728.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.44728.log.gz 2024-09-11T09:16:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48477.log 2024-09-11T09:16:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59372.log 2024-09-11T09:16:41.334 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53103.log: /var/log/ceph/cluster2-client.mirror.48477.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53103.log.gz 2024-09-11T09:16:41.335 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40095.log 2024-09-11T09:16:41.336 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59372.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48477.log.gz 2024-09-11T09:16:41.337 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59372.log.gz 2024-09-11T09:16:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49975.log 2024-09-11T09:16:41.338 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49348.log 2024-09-11T09:16:41.339 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40095.log: /var/log/ceph/cluster1-client.mirror.49975.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.49975.log.gz 2024-09-11T09:16:41.348 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64674.log 2024-09-11T09:16:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49348.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49348.log.gz 2024-09-11T09:16:41.349 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.63162.log 2024-09-11T09:16:41.350 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64674.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64674.log.gz 2024-09-11T09:16:41.351 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38583.log 2024-09-11T09:16:41.356 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.1.63162.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.63162.log.gz 2024-09-11T09:16:41.357 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35015.log 2024-09-11T09:16:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38583.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38583.log.gz 2024-09-11T09:16:41.358 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60285.log 2024-09-11T09:16:41.358 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40188.log.gz 2024-09-11T09:16:41.359 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35015.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43745.log 2024-09-11T09:16:41.359 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60285.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.35015.log.gz 2024-09-11T09:16:41.360 INFO:teuthology.orchestra.run.smithi077.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60285.log.gz 2024-09-11T09:16:41.360 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35265.log 2024-09-11T09:16:41.360 INFO:teuthology.orchestra.run.smithi073.stderr:/var/log/ceph/cluster1-osd.1.log: 91.2% -- replaced with /var/log/ceph/cluster1-mgr.x.log.gz 2024-09-11T09:16:41.361 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40210.log 2024-09-11T09:16:41.361 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43745.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43745.log.gz 2024-09-11T09:16:41.361 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35265.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61455.log 2024-09-11T09:16:41.361 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40210.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40210.log.gz 2024-09-11T09:16:41.363 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57165.log 2024-09-11T09:16:41.363 INFO:teuthology.orchestra.run.smithi077.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35265.log.gz 2024-09-11T09:16:41.363 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61455.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61455.log.gz 2024-09-11T09:16:41.363 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30395.log 2024-09-11T09:16:41.364 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57165.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57165.log.gz 2024-09-11T09:16:41.366 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32324.log 2024-09-11T09:16:41.366 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30395.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54609.log 2024-09-11T09:16:41.366 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30395.log.gz 2024-09-11T09:16:41.366 INFO:teuthology.orchestra.run.smithi073.stderr: 91.2% -- replaced with /var/log/ceph/cluster2-mgr.x.log.gz 2024-09-11T09:16:41.367 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32324.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44981.log 2024-09-11T09:16:41.367 INFO:teuthology.orchestra.run.smithi077.stderr: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.32324.log.gz 2024-09-11T09:16:41.367 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54609.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54496.log 2024-09-11T09:16:41.368 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44981.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44981.log.gz 2024-09-11T09:16:41.368 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55839.log 2024-09-11T09:16:41.368 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54609.log.gz 2024-09-11T09:16:41.369 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54496.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37273.log 2024-09-11T09:16:41.369 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55839.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54496.log.gz 2024-09-11T09:16:41.369 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37449.log 2024-09-11T09:16:41.370 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37273.log: 89.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37273.log.gz 2024-09-11T09:16:41.370 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38443.log 2024-09-11T09:16:41.371 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55839.log.gz 2024-09-11T09:16:41.371 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37449.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42360.log 2024-09-11T09:16:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38443.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38443.log.gz 2024-09-11T09:16:41.372 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37965.log 2024-09-11T09:16:41.376 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42360.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37449.log.gz 2024-09-11T09:16:41.376 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61275.log 2024-09-11T09:16:41.382 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37965.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37965.log.gz 2024-09-11T09:16:41.392 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.1.log 2024-09-11T09:16:41.393 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61275.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61275.log.gz 2024-09-11T09:16:41.393 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54026.log 2024-09-11T09:16:41.400 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-osd.1.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36539.log 2024-09-11T09:16:41.408 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54026.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30571.log 2024-09-11T09:16:41.409 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.54026.log.gz 2024-09-11T09:16:41.409 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36539.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36539.log.gz 2024-09-11T09:16:41.424 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52549.log 2024-09-11T09:16:41.424 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.30571.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30571.log.gz 2024-09-11T09:16:41.425 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35909.log 2024-09-11T09:16:41.425 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52549.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52549.log.gz 2024-09-11T09:16:41.429 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33901.log.gz 2024-09-11T09:16:41.430 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46735.log 2024-09-11T09:16:41.430 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35909.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35909.log.gz 2024-09-11T09:16:41.431 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47205.log 2024-09-11T09:16:41.433 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46735.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46735.log.gz 2024-09-11T09:16:41.433 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47225.log 2024-09-11T09:16:41.435 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47205.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47205.log.gz 2024-09-11T09:16:41.436 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56311.log 2024-09-11T09:16:41.437 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47225.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47225.log.gz 2024-09-11T09:16:41.438 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60712.log 2024-09-11T09:16:41.439 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56311.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.56311.log.gz 2024-09-11T09:16:41.439 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51902.log 2024-09-11T09:16:41.440 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60712.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60712.log.gz 2024-09-11T09:16:41.440 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30796.log 2024-09-11T09:16:41.441 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51902.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.51902.log.gz 2024-09-11T09:16:41.442 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34363.log 2024-09-11T09:16:41.443 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.30796.log: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.2.40811.log.gz 2024-09-11T09:16:41.447 INFO:teuthology.orchestra.run.smithi077.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30796.log.gz 2024-09-11T09:16:41.448 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.43567.log 2024-09-11T09:16:41.448 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34363.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57827.log 2024-09-11T09:16:41.449 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.1.43567.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.34363.log.gz 2024-09-11T09:16:41.449 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56176.log 2024-09-11T09:16:41.450 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57827.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57827.log.gz 2024-09-11T09:16:41.451 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42953.log 2024-09-11T09:16:41.452 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56176.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.56176.log.gz 2024-09-11T09:16:41.452 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65909.log 2024-09-11T09:16:41.453 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42953.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42953.log.gz 2024-09-11T09:16:41.453 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37945.log 2024-09-11T09:16:41.454 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65909.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65909.log.gz 2024-09-11T09:16:41.455 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64781.log 2024-09-11T09:16:41.456 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37945.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.1.43567.log.gz 2024-09-11T09:16:41.460 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37945.log.gz 2024-09-11T09:16:41.460 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49574.log 2024-09-11T09:16:41.461 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53689.log 2024-09-11T09:16:41.461 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64781.log: /var/log/ceph/cluster2-client.mirror.49574.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64781.log.gz 2024-09-11T09:16:41.462 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59682.log 2024-09-11T09:16:41.462 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.49574.log.gz 2024-09-11T09:16:41.464 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53689.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53689.log.gz 2024-09-11T09:16:41.464 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58918.log 2024-09-11T09:16:41.464 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58219.log 2024-09-11T09:16:41.465 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59682.log: /var/log/ceph/cluster2-client.mirror.58918.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58918.log.gz 2024-09-11T09:16:41.467 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59682.log.gz 2024-09-11T09:16:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65649.log 2024-09-11T09:16:41.467 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38753.log 2024-09-11T09:16:41.469 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58219.log: /var/log/ceph/cluster2-client.mirror.65649.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58219.log.gz 2024-09-11T09:16:41.470 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58116.log 2024-09-11T09:16:41.470 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65649.log.gz 2024-09-11T09:16:41.484 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38753.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59310.log 2024-09-11T09:16:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58116.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58116.log.gz 2024-09-11T09:16:41.486 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52131.log 2024-09-11T09:16:41.488 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59310.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59310.log.gz 2024-09-11T09:16:41.489 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31409.log 2024-09-11T09:16:41.489 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52131.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52131.log.gz 2024-09-11T09:16:41.490 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59124.log 2024-09-11T09:16:41.491 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31409.log: 55.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31409.log.gz 2024-09-11T09:16:41.491 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63934.log 2024-09-11T09:16:41.493 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59124.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59124.log.gz 2024-09-11T09:16:41.494 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37558.log 2024-09-11T09:16:41.501 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63934.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38203.log 2024-09-11T09:16:41.505 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37558.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41582.log.gz 2024-09-11T09:16:41.506 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37558.log.gz 2024-09-11T09:16:41.520 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59662.log 2024-09-11T09:16:41.521 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38203.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38203.log.gz 2024-09-11T09:16:41.522 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50096.log 2024-09-11T09:16:41.523 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59662.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59662.log.gz 2024-09-11T09:16:41.523 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52107.log 2024-09-11T09:16:41.524 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50096.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50096.log.gz 2024-09-11T09:16:41.524 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64526.log 2024-09-11T09:16:41.525 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52107.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52107.log.gz 2024-09-11T09:16:41.528 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58980.log 2024-09-11T09:16:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64526.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64526.log.gz 2024-09-11T09:16:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50038.log 2024-09-11T09:16:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58980.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58980.log.gz 2024-09-11T09:16:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37105.log 2024-09-11T09:16:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50038.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50038.log.gz 2024-09-11T09:16:41.530 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-mon.a.log 2024-09-11T09:16:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37105.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37105.log.gz 2024-09-11T09:16:41.531 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61095.log 2024-09-11T09:16:41.544 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-mon.a.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54351.log 2024-09-11T09:16:41.545 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61095.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61095.log.gz 2024-09-11T09:16:41.560 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39569.log 2024-09-11T09:16:41.561 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54351.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.54351.log.gz 2024-09-11T09:16:41.572 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34923.log 2024-09-11T09:16:41.573 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39569.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39569.log.gz 2024-09-11T09:16:41.584 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.30796.log 2024-09-11T09:16:41.585 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34923.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34923.log.gz 2024-09-11T09:16:41.592 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58835.log 2024-09-11T09:16:41.600 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.30796.log: 93.7%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52638.log 2024-09-11T09:16:41.601 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58835.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58835.log.gz 2024-09-11T09:16:41.608 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34967.log 2024-09-11T09:16:41.610 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52638.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52638.log.gz 2024-09-11T09:16:41.612 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.30796.log.gz 2024-09-11T09:16:41.616 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44959.log 2024-09-11T09:16:41.617 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34967.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.34967.log.gz 2024-09-11T09:16:41.620 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39690.log.gz 2024-09-11T09:16:41.621 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45509.log 2024-09-11T09:16:41.622 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44959.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44959.log.gz 2024-09-11T09:16:41.623 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58033.log 2024-09-11T09:16:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45509.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.45509.log.gz 2024-09-11T09:16:41.624 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48124.log 2024-09-11T09:16:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58033.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58033.log.gz 2024-09-11T09:16:41.627 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32694.log 2024-09-11T09:16:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48124.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.48124.log.gz 2024-09-11T09:16:41.628 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51738.log 2024-09-11T09:16:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.32694.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.32694.log.gz 2024-09-11T09:16:41.629 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64584.log 2024-09-11T09:16:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51738.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51738.log.gz 2024-09-11T09:16:41.632 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40688.log 2024-09-11T09:16:41.633 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64584.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64584.log.gz 2024-09-11T09:16:41.634 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60870.log 2024-09-11T09:16:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40688.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40688.log.gz 2024-09-11T09:16:41.635 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57227.log 2024-09-11T09:16:41.636 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60870.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60870.log.gz 2024-09-11T09:16:41.636 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58794.log 2024-09-11T09:16:41.639 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57227.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57227.log.gz 2024-09-11T09:16:41.640 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50703.log 2024-09-11T09:16:41.640 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40710.log.gz 2024-09-11T09:16:41.640 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58794.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58794.log.gz 2024-09-11T09:16:41.640 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42281.log 2024-09-11T09:16:41.641 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30546.log 2024-09-11T09:16:41.642 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50703.log: /var/log/ceph/cluster2-client.mirror.42281.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50703.log.gz 2024-09-11T09:16:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50658.log 2024-09-11T09:16:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.30546.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30546.log.gz 2024-09-11T09:16:41.643 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50120.log 2024-09-11T09:16:41.644 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50658.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.50658.log.gz 2024-09-11T09:16:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37492.log 2024-09-11T09:16:41.645 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50120.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50120.log.gz 2024-09-11T09:16:41.646 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64113.log 2024-09-11T09:16:41.651 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37492.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37492.log.gz 2024-09-11T09:16:41.651 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55798.log 2024-09-11T09:16:41.654 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64113.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64113.log.gz 2024-09-11T09:16:41.654 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.32955.log 2024-09-11T09:16:41.655 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55798.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.55798.log.gz 2024-09-11T09:16:41.655 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52218.log 2024-09-11T09:16:41.658 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.32955.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.0.32955.log.gz 2024-09-11T09:16:41.658 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42797.log 2024-09-11T09:16:41.659 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52218.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.52218.log.gz 2024-09-11T09:16:41.659 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42459.log 2024-09-11T09:16:41.667 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42797.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40095.log.gz 2024-09-11T09:16:41.672 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63006.log 2024-09-11T09:16:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42459.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42459.log.gz 2024-09-11T09:16:41.673 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46305.log 2024-09-11T09:16:41.674 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63006.log: 83.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63006.log.gz 2024-09-11T09:16:41.675 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35447.log 2024-09-11T09:16:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46305.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46305.log.gz 2024-09-11T09:16:41.677 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55907.log 2024-09-11T09:16:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35447.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35447.log.gz 2024-09-11T09:16:41.678 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45758.log 2024-09-11T09:16:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55907.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55907.log.gz 2024-09-11T09:16:41.680 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50859.log 2024-09-11T09:16:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45758.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45758.log.gz 2024-09-11T09:16:41.681 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52328.log 2024-09-11T09:16:41.682 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50859.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50859.log.gz 2024-09-11T09:16:41.682 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.29998.log 2024-09-11T09:16:41.683 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52328.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52328.log.gz 2024-09-11T09:16:41.684 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56378.log 2024-09-11T09:16:41.684 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.29998.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.29998.log.gz 2024-09-11T09:16:41.685 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50076.log 2024-09-11T09:16:41.685 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56378.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56378.log.gz 2024-09-11T09:16:41.686 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50361.log 2024-09-11T09:16:41.687 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50076.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50076.log.gz 2024-09-11T09:16:41.688 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44386.log 2024-09-11T09:16:41.688 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50361.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38753.log.gz 2024-09-11T09:16:41.689 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50361.log.gz 2024-09-11T09:16:41.689 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32787.log 2024-09-11T09:16:41.689 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52352.log 2024-09-11T09:16:41.691 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44386.log: /var/log/ceph/cluster2-client.mirror.32787.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.44386.log.gz 2024-09-11T09:16:41.691 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58334.log 2024-09-11T09:16:41.692 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52352.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52352.log.gz 2024-09-11T09:16:41.692 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52841.log 2024-09-11T09:16:41.693 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58334.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58334.log.gz 2024-09-11T09:16:41.694 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57744.log 2024-09-11T09:16:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52841.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.32787.log.gz 2024-09-11T09:16:41.696 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52841.log.gz 2024-09-11T09:16:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30495.log 2024-09-11T09:16:41.696 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42360.log.gz 2024-09-11T09:16:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31112.log 2024-09-11T09:16:41.696 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57744.log: /var/log/ceph/cluster2-client.admin.30495.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35127.log 2024-09-11T09:16:41.696 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30495.log.gz 2024-09-11T09:16:41.698 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57744.log.gz 2024-09-11T09:16:41.698 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose/var/log/ceph/cluster1-client.admin.31112.log: -- /var/log/ceph/cluster2-client.mirror.34790.log 2024-09-11T09:16:41.698 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31112.log.gz 2024-09-11T09:16:41.699 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32413.log 2024-09-11T09:16:41.699 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35127.log: /var/log/ceph/cluster2-client.mirror.34790.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57041.log 2024-09-11T09:16:41.699 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32413.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32413.log.gz 2024-09-11T09:16:41.699 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61365.log 2024-09-11T09:16:41.699 INFO:teuthology.orchestra.run.smithi077.stderr: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.34790.log.gz 2024-09-11T09:16:41.700 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57041.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42318.log 2024-09-11T09:16:41.700 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61365.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.61365.log.gz 2024-09-11T09:16:41.700 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31477.log 2024-09-11T09:16:41.701 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42318.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57041.log.gz 2024-09-11T09:16:41.701 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38094.log 2024-09-11T09:16:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31477.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42318.log.gz 2024-09-11T09:16:41.702 INFO:teuthology.orchestra.run.smithi077.stderr: 58.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31477.log.gz 2024-09-11T09:16:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39014.log 2024-09-11T09:16:41.702 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5%gzip -- replaced with /var/log/ceph/cluster2-client.mirror.35127.log.gz 2024-09-11T09:16:41.702 INFO:teuthology.orchestra.run.smithi077.stderr: -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31494.log 2024-09-11T09:16:41.702 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.38094.log: /var/log/ceph/cluster2-client.mirror.39014.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46461.log 2024-09-11T09:16:41.703 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31494.log: 54.2% -- replaced with /var/log/ceph/cluster2-client.mirror.31494.log.gz 2024-09-11T09:16:41.703 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43545.log 2024-09-11T09:16:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46461.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46461.log.gz 2024-09-11T09:16:41.704 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33110.log 2024-09-11T09:16:41.707 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43545.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38094.log.gz 2024-09-11T09:16:41.708 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48613.log 2024-09-11T09:16:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33110.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33110.log.gz 2024-09-11T09:16:41.709 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50835.log 2024-09-11T09:16:41.710 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48613.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48613.log.gz 2024-09-11T09:16:41.710 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64651.log 2024-09-11T09:16:41.711 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50835.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.50835.log.gz 2024-09-11T09:16:41.711 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42827.log 2024-09-11T09:16:41.712 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64651.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64651.log.gz 2024-09-11T09:16:41.713 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42738.log 2024-09-11T09:16:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.42827.log: 95.3% -- replaced with /var/log/ceph/cluster1-client.mirror.42827.log.gz 2024-09-11T09:16:41.717 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66497.log 2024-09-11T09:16:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42738.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.42738.log.gz 2024-09-11T09:16:41.718 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34109.log 2024-09-11T09:16:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66497.log: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.66497.log.gz 2024-09-11T09:16:41.719 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65288.log 2024-09-11T09:16:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34109.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34109.log.gz 2024-09-11T09:16:41.720 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45465.log 2024-09-11T09:16:41.721 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65288.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65288.log.gz 2024-09-11T09:16:41.722 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54897.log 2024-09-11T09:16:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45465.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45465.log.gz 2024-09-11T09:16:41.723 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55750.log 2024-09-11T09:16:41.726 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54897.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54897.log.gz 2024-09-11T09:16:41.726 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48076.log 2024-09-11T09:16:41.727 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55750.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55750.log.gz 2024-09-11T09:16:41.727 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42338.log 2024-09-11T09:16:41.728 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48076.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48076.log.gz 2024-09-11T09:16:41.728 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61432.log 2024-09-11T09:16:41.733 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42338.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42338.log.gz 2024-09-11T09:16:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63275.log 2024-09-11T09:16:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61432.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61432.log.gz 2024-09-11T09:16:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37793.log 2024-09-11T09:16:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63275.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.63275.log.gz 2024-09-11T09:16:41.734 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38563.log 2024-09-11T09:16:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37793.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37793.log.gz 2024-09-11T09:16:41.738 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53825.log 2024-09-11T09:16:41.739 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38563.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38563.log.gz 2024-09-11T09:16:41.739 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51649.log 2024-09-11T09:16:41.740 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53825.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53825.log.gz 2024-09-11T09:16:41.740 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45782.log 2024-09-11T09:16:41.741 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51649.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.51649.log.gz 2024-09-11T09:16:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63834.log 2024-09-11T09:16:41.742 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45782.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45782.log.gz 2024-09-11T09:16:41.743 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65995.log 2024-09-11T09:16:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63834.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.63834.log.gz 2024-09-11T09:16:41.744 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36741.log 2024-09-11T09:16:41.745 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65995.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65995.log.gz 2024-09-11T09:16:41.745 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39648.log 2024-09-11T09:16:41.747 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36741.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36741.log.gz 2024-09-11T09:16:41.747 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64349.log 2024-09-11T09:16:41.749 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39648.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39648.log.gz 2024-09-11T09:16:41.749 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61770.log 2024-09-11T09:16:41.750 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64349.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64349.log.gz 2024-09-11T09:16:41.751 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35375.log 2024-09-11T09:16:41.751 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61770.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61770.log.gz 2024-09-11T09:16:41.752 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41965.log 2024-09-11T09:16:41.753 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35375.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.35375.log.gz 2024-09-11T09:16:41.753 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44778.log 2024-09-11T09:16:41.754 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41965.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41965.log.gz 2024-09-11T09:16:41.754 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43486.log 2024-09-11T09:16:41.755 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44778.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44778.log.gz 2024-09-11T09:16:41.756 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31550.log 2024-09-11T09:16:41.756 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43486.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43486.log.gz 2024-09-11T09:16:41.757 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64859.log 2024-09-11T09:16:41.758 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31550.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.31550.log.gz 2024-09-11T09:16:41.758 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53552.log 2024-09-11T09:16:41.759 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64859.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64859.log.gz 2024-09-11T09:16:41.760 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44272.log 2024-09-11T09:16:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53552.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53552.log.gz 2024-09-11T09:16:41.761 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53424.log 2024-09-11T09:16:41.762 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44272.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44272.log.gz 2024-09-11T09:16:41.762 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52376.log 2024-09-11T09:16:41.765 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53424.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53424.log.gz 2024-09-11T09:16:41.765 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48902.log 2024-09-11T09:16:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52376.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52376.log.gz 2024-09-11T09:16:41.766 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38864.log 2024-09-11T09:16:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48902.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48902.log.gz 2024-09-11T09:16:41.767 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66813.log 2024-09-11T09:16:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38864.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.38864.log.gz 2024-09-11T09:16:41.768 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36990.log 2024-09-11T09:16:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.66813.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66813.log.gz 2024-09-11T09:16:41.769 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52280.log 2024-09-11T09:16:41.770 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36990.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36990.log.gz 2024-09-11T09:16:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62040.log 2024-09-11T09:16:41.771 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52280.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52280.log.gz 2024-09-11T09:16:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51531.log 2024-09-11T09:16:41.772 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.62040.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.62040.log.gz 2024-09-11T09:16:41.773 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65865.log 2024-09-11T09:16:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51531.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51531.log.gz 2024-09-11T09:16:41.774 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45188.log 2024-09-11T09:16:41.775 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65865.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65865.log.gz 2024-09-11T09:16:41.776 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35351.log 2024-09-11T09:16:41.778 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45188.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.45188.log.gz 2024-09-11T09:16:41.778 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52660.log 2024-09-11T09:16:41.779 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35351.log: 88.7% -- replaced with /var/log/ceph/cluster1-client.mirror.35351.log.gz 2024-09-11T09:16:41.779 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61860.log 2024-09-11T09:16:41.781 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52660.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52660.log.gz 2024-09-11T09:16:41.781 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57434.log 2024-09-11T09:16:41.782 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61860.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61860.log.gz 2024-09-11T09:16:41.782 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58504.log 2024-09-11T09:16:41.783 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57434.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57434.log.gz 2024-09-11T09:16:41.783 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37707.log 2024-09-11T09:16:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58504.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58504.log.gz 2024-09-11T09:16:41.786 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63252.log 2024-09-11T09:16:41.792 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37707.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37707.log.gz 2024-09-11T09:16:41.792 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51947.log 2024-09-11T09:16:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63252.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.63252.log.gz 2024-09-11T09:16:41.793 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63934.log.gz 2024-09-11T09:16:41.793 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35974.log 2024-09-11T09:16:41.794 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58814.log 2024-09-11T09:16:41.796 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51947.log: /var/log/ceph/cluster1-client.mirror.35974.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.51947.log.gz 2024-09-11T09:16:41.796 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61117.log 2024-09-11T09:16:41.798 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58814.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58814.log.gz 2024-09-11T09:16:41.799 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36017.log 2024-09-11T09:16:41.800 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4%/var/log/ceph/cluster1-client.mirror.61117.log: -- replaced with /var/log/ceph/cluster1-client.mirror.35974.log.gz 2024-09-11T09:16:41.800 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61117.log.gz 2024-09-11T09:16:41.800 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57930.log 2024-09-11T09:16:41.800 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36017.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33066.log 2024-09-11T09:16:41.801 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57930.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57930.log.gz 2024-09-11T09:16:41.801 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63417.log 2024-09-11T09:16:41.802 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33066.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33066.log.gz 2024-09-11T09:16:41.803 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31533.log 2024-09-11T09:16:41.806 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63417.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36017.log.gz 2024-09-11T09:16:41.806 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37730.log 2024-09-11T09:16:41.806 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31533.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.31533.log.gz 2024-09-11T09:16:41.807 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66096.log 2024-09-11T09:16:41.812 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37730.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37730.log.gz 2024-09-11T09:16:41.813 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42518.log 2024-09-11T09:16:41.828 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66096.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56565.log 2024-09-11T09:16:41.840 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42518.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49027.log 2024-09-11T09:16:41.841 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56565.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56565.log.gz 2024-09-11T09:16:41.856 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50907.log 2024-09-11T09:16:41.857 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49027.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49027.log.gz 2024-09-11T09:16:41.864 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36200.log 2024-09-11T09:16:41.865 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50907.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50907.log.gz 2024-09-11T09:16:41.872 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60915.log 2024-09-11T09:16:41.873 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36200.log: 93.9% -- replaced with /var/log/ceph/cluster1-client.mirror.36200.log.gz 2024-09-11T09:16:41.880 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41619.log 2024-09-11T09:16:41.881 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60915.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60915.log.gz 2024-09-11T09:16:41.888 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59538.log 2024-09-11T09:16:41.889 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41619.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41619.log.gz 2024-09-11T09:16:41.900 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59992.log 2024-09-11T09:16:41.901 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59538.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59538.log.gz 2024-09-11T09:16:41.901 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42797.log.gz 2024-09-11T09:16:41.902 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39727.log 2024-09-11T09:16:41.902 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59992.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59992.log.gz 2024-09-11T09:16:41.903 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59970.log 2024-09-11T09:16:41.904 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39727.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39727.log.gz 2024-09-11T09:16:41.904 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58752.log 2024-09-11T09:16:41.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59970.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.59970.log.gz 2024-09-11T09:16:41.905 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48165.log 2024-09-11T09:16:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58752.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58752.log.gz 2024-09-11T09:16:41.908 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56498.log 2024-09-11T09:16:41.910 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48165.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48165.log.gz 2024-09-11T09:16:41.910 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67034.log 2024-09-11T09:16:41.911 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56498.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56498.log.gz 2024-09-11T09:16:41.912 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53379.log 2024-09-11T09:16:41.912 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.67034.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67034.log.gz 2024-09-11T09:16:41.913 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41190.log 2024-09-11T09:16:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53379.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53379.log.gz 2024-09-11T09:16:41.914 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57951.log 2024-09-11T09:16:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41190.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41190.log.gz 2024-09-11T09:16:41.915 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50293.log 2024-09-11T09:16:41.916 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57951.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57951.log.gz 2024-09-11T09:16:41.917 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61477.log 2024-09-11T09:16:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50293.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.50293.log.gz 2024-09-11T09:16:41.918 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34337.log 2024-09-11T09:16:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61477.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61477.log.gz 2024-09-11T09:16:41.919 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33086.log 2024-09-11T09:16:41.920 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.34337.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.admin.34337.log.gz 2024-09-11T09:16:41.921 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.36200.log 2024-09-11T09:16:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33086.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33086.log.gz 2024-09-11T09:16:41.922 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33653.log 2024-09-11T09:16:41.934 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.36200.log: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33522.log 2024-09-11T09:16:41.935 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33653.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33653.log.gz 2024-09-11T09:16:41.935 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.0.36200.log.gz 2024-09-11T09:16:41.952 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33270.log 2024-09-11T09:16:41.953 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33522.log: 81.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33522.log.gz 2024-09-11T09:16:41.976 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43861.log 2024-09-11T09:16:41.982 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33270.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33270.log.gz 2024-09-11T09:16:41.996 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35307.log 2024-09-11T09:16:41.997 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43861.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43861.log.gz 2024-09-11T09:16:42.004 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35212.log 2024-09-11T09:16:42.005 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35307.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.35307.log.gz 2024-09-11T09:16:42.011 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64903.log 2024-09-11T09:16:42.011 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.35212.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.35212.log.gz 2024-09-11T09:16:42.011 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.43545.log.gz 2024-09-11T09:16:42.012 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35997.log 2024-09-11T09:16:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64903.log: 80.5% -- replaced with /var/log/ceph/cluster2-client.mirror.64903.log.gz 2024-09-11T09:16:42.013 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33749.log 2024-09-11T09:16:42.018 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35997.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35997.log.gz 2024-09-11T09:16:42.019 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56586.log 2024-09-11T09:16:42.019 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33749.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.33749.log.gz 2024-09-11T09:16:42.020 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42397.log 2024-09-11T09:16:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56586.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56586.log.gz 2024-09-11T09:16:42.021 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45734.log 2024-09-11T09:16:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42397.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42397.log.gz 2024-09-11T09:16:42.022 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56917.log 2024-09-11T09:16:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45734.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.45734.log.gz 2024-09-11T09:16:42.023 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50970.log 2024-09-11T09:16:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56917.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56917.log.gz 2024-09-11T09:16:42.026 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46777.log 2024-09-11T09:16:42.027 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50970.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.50970.log.gz 2024-09-11T09:16:42.028 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37253.log 2024-09-11T09:16:42.029 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46777.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46777.log.gz 2024-09-11T09:16:42.029 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32807.log 2024-09-11T09:16:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37253.log: 90.2% -- replaced with /var/log/ceph/cluster2-client.mirror.37253.log.gz 2024-09-11T09:16:42.030 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49642.log 2024-09-11T09:16:42.036 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.32807.log: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.32807.log.gz 2024-09-11T09:16:42.037 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60195.log 2024-09-11T09:16:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49642.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.49642.log.gz 2024-09-11T09:16:42.038 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39106.log 2024-09-11T09:16:42.039 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60195.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60195.log.gz 2024-09-11T09:16:42.039 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33227.log 2024-09-11T09:16:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39106.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.39106.log.gz 2024-09-11T09:16:42.040 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47006.log 2024-09-11T09:16:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33227.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.33227.log.gz 2024-09-11T09:16:42.046 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45851.log 2024-09-11T09:16:42.047 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47006.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47006.log.gz 2024-09-11T09:16:42.048 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48661.log 2024-09-11T09:16:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45851.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45851.log.gz 2024-09-11T09:16:42.049 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60082.log 2024-09-11T09:16:42.050 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48661.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48661.log.gz 2024-09-11T09:16:42.050 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65201.log 2024-09-11T09:16:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60082.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60082.log.gz 2024-09-11T09:16:42.051 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52397.log 2024-09-11T09:16:42.052 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65201.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65201.log.gz 2024-09-11T09:16:42.053 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52976.log 2024-09-11T09:16:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52397.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52397.log.gz 2024-09-11T09:16:42.055 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45951.log 2024-09-11T09:16:42.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52976.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52976.log.gz 2024-09-11T09:16:42.056 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54255.log 2024-09-11T09:16:42.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45951.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45951.log.gz 2024-09-11T09:16:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46642.log 2024-09-11T09:16:42.058 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54255.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54255.log.gz 2024-09-11T09:16:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65474.log 2024-09-11T09:16:42.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46642.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46642.log.gz 2024-09-11T09:16:42.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64072.log 2024-09-11T09:16:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65415.log 2024-09-11T09:16:42.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64072.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64072.log.gz 2024-09-11T09:16:42.081 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42281.log.gz 2024-09-11T09:16:42.082 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36942.log 2024-09-11T09:16:42.082 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65415.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65415.log.gz 2024-09-11T09:16:42.083 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34943.log 2024-09-11T09:16:42.084 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36942.log: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.36942.log.gz 2024-09-11T09:16:42.084 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51331.log 2024-09-11T09:16:42.085 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34943.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.34943.log.gz 2024-09-11T09:16:42.085 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67060.log 2024-09-11T09:16:42.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51331.log: 94.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51331.log.gz 2024-09-11T09:16:42.087 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43266.log 2024-09-11T09:16:42.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.67060.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67060.log.gz 2024-09-11T09:16:42.088 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43762.log 2024-09-11T09:16:42.089 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43266.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43266.log.gz 2024-09-11T09:16:42.089 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36918.log 2024-09-11T09:16:42.090 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43762.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43762.log.gz 2024-09-11T09:16:42.091 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48589.log 2024-09-11T09:16:42.092 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36918.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36918.log.gz 2024-09-11T09:16:42.092 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.33430.log 2024-09-11T09:16:42.093 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48589.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48589.log.gz 2024-09-11T09:16:42.093 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43444.log 2024-09-11T09:16:42.097 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.33430.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.33430.log.gz 2024-09-11T09:16:42.098 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43782.log 2024-09-11T09:16:42.098 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43444.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43444.log.gz 2024-09-11T09:16:42.099 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43673.log 2024-09-11T09:16:42.099 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43782.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43782.log.gz 2024-09-11T09:16:42.100 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59186.log 2024-09-11T09:16:42.101 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43673.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43673.log.gz 2024-09-11T09:16:42.102 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51281.log 2024-09-11T09:16:42.103 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59186.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59186.log.gz 2024-09-11T09:16:42.104 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30161.log 2024-09-11T09:16:42.104 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51281.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51281.log.gz 2024-09-11T09:16:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57496.log 2024-09-11T09:16:42.105 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30161.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30161.log.gz 2024-09-11T09:16:42.106 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39426.log 2024-09-11T09:16:42.106 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57496.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57496.log.gz 2024-09-11T09:16:42.107 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59947.log 2024-09-11T09:16:42.107 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39426.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39426.log.gz 2024-09-11T09:16:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60937.log 2024-09-11T09:16:42.108 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59947.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59947.log.gz 2024-09-11T09:16:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55232.log 2024-09-11T09:16:42.109 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60937.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60937.log.gz 2024-09-11T09:16:42.110 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.audit.log 2024-09-11T09:16:42.110 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55232.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.55232.log.gz 2024-09-11T09:16:42.111 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45975.log 2024-09-11T09:16:42.114 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2.audit.log: 92.6% -- replaced with /var/log/ceph/cluster2.audit.log.gz 2024-09-11T09:16:42.115 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55025.log 2024-09-11T09:16:42.116 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45975.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45975.log.gz 2024-09-11T09:16:42.116 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66410.log 2024-09-11T09:16:42.117 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55025.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55025.log.gz 2024-09-11T09:16:42.117 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64284.log 2024-09-11T09:16:42.118 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.66410.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.66410.log.gz 2024-09-11T09:16:42.118 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47377.log 2024-09-11T09:16:42.130 INFO:teuthology.orchestra.run.smithi073.stderr: 90.3% -- replaced with /var/log/ceph/cluster1-mon.a.log.gz 2024-09-11T09:16:42.132 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64284.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55726.log 2024-09-11T09:16:42.133 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47377.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47377.log.gz 2024-09-11T09:16:42.138 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.66096.log.gz 2024-09-11T09:16:42.138 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34221.log 2024-09-11T09:16:42.139 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55726.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55726.log.gz 2024-09-11T09:16:42.140 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48100.log 2024-09-11T09:16:42.145 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34221.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.34221.log.gz 2024-09-11T09:16:42.146 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44617.log 2024-09-11T09:16:42.147 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48100.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48100.log.gz 2024-09-11T09:16:42.147 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32584.log 2024-09-11T09:16:42.148 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44617.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44617.log.gz 2024-09-11T09:16:42.148 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65181.log 2024-09-11T09:16:42.150 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32584.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32584.log.gz 2024-09-11T09:16:42.150 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60622.log 2024-09-11T09:16:42.151 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65181.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65181.log.gz 2024-09-11T09:16:42.152 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60645.log 2024-09-11T09:16:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60622.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60622.log.gz 2024-09-11T09:16:42.153 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41795.log 2024-09-11T09:16:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60645.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60645.log.gz 2024-09-11T09:16:42.154 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56044.log 2024-09-11T09:16:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41795.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41795.log.gz 2024-09-11T09:16:42.155 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33725.log 2024-09-11T09:16:42.157 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56044.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56044.log.gz 2024-09-11T09:16:42.158 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39710.log 2024-09-11T09:16:42.159 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33725.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.33725.log.gz 2024-09-11T09:16:42.159 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48748.log 2024-09-11T09:16:42.160 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39710.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39710.log.gz 2024-09-11T09:16:42.160 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58157.log 2024-09-11T09:16:42.161 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48748.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48748.log.gz 2024-09-11T09:16:42.162 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2.log 2024-09-11T09:16:42.164 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58157.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58157.log.gz 2024-09-11T09:16:42.165 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66930.log 2024-09-11T09:16:42.167 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39014.log.gz 2024-09-11T09:16:42.170 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster2.log.gz 2024-09-11T09:16:42.170 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50723.log 2024-09-11T09:16:42.171 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43054.log 2024-09-11T09:16:42.171 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.66930.log: /var/log/ceph/cluster1-client.mirror.50723.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66930.log.gz 2024-09-11T09:16:42.172 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49372.log 2024-09-11T09:16:42.173 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43054.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50723.log.gz 2024-09-11T09:16:42.184 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57103.log 2024-09-11T09:16:42.185 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49372.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49372.log.gz 2024-09-11T09:16:42.185 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55861.log 2024-09-11T09:16:42.188 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57103.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57103.log.gz 2024-09-11T09:16:42.188 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32907.log 2024-09-11T09:16:42.190 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55861.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55861.log.gz 2024-09-11T09:16:42.190 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43012.log 2024-09-11T09:16:42.191 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32907.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32907.log.gz 2024-09-11T09:16:42.191 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40036.log 2024-09-11T09:16:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43012.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43012.log.gz 2024-09-11T09:16:42.193 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53914.log 2024-09-11T09:16:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40036.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40036.log.gz 2024-09-11T09:16:42.194 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60825.log 2024-09-11T09:16:42.196 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53914.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53914.log.gz 2024-09-11T09:16:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37146.log 2024-09-11T09:16:42.197 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60825.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60825.log.gz 2024-09-11T09:16:42.198 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45871.log 2024-09-11T09:16:42.203 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37146.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37146.log.gz 2024-09-11T09:16:42.204 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50192.log 2024-09-11T09:16:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45871.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45871.log.gz 2024-09-11T09:16:42.206 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43249.log 2024-09-11T09:16:42.207 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50192.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.50192.log.gz 2024-09-11T09:16:42.207 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.63417.log.gz 2024-09-11T09:16:42.208 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57889.log 2024-09-11T09:16:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30469.log 2024-09-11T09:16:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43249.log: /var/log/ceph/cluster2-client.mirror.57889.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43249.log.gz 2024-09-11T09:16:42.209 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57889.log.gz 2024-09-11T09:16:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57310.log 2024-09-11T09:16:42.209 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50682.log 2024-09-11T09:16:42.210 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30469.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30469.log.gz 2024-09-11T09:16:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57310.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49879.log 2024-09-11T09:16:42.211 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1%/var/log/ceph/cluster1-client.mirror.50682.log: -- replaced with /var/log/ceph/cluster2-client.mirror.57310.log.gz 2024-09-11T09:16:42.211 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0% -- replaced with /var/log/ceph/cluster1-client.mirror.50682.log.gz 2024-09-11T09:16:42.211 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63086.log 2024-09-11T09:16:42.212 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61410.log 2024-09-11T09:16:42.212 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49879.log: /var/log/ceph/cluster1-client.mirror.63086.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49879.log.gz 2024-09-11T09:16:42.212 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63086.log.gz 2024-09-11T09:16:42.212 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.39791.log 2024-09-11T09:16:42.213 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49164.log 2024-09-11T09:16:42.213 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61410.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61410.log.gz 2024-09-11T09:16:42.213 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.39791.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57021.log 2024-09-11T09:16:42.215 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49164.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.49164.log.gz 2024-09-11T09:16:42.215 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.34310.log 2024-09-11T09:16:42.216 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57021.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57021.log.gz 2024-09-11T09:16:42.217 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43693.log 2024-09-11T09:16:42.218 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.34310.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.34310.log.gz 2024-09-11T09:16:42.218 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67008.log 2024-09-11T09:16:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43693.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43693.log.gz 2024-09-11T09:16:42.219 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48432.log 2024-09-11T09:16:42.220 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.67008.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67008.log.gz 2024-09-11T09:16:42.220 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48456.log 2024-09-11T09:16:42.221 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48432.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48432.log.gz 2024-09-11T09:16:42.222 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38343.log 2024-09-11T09:16:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48456.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48456.log.gz 2024-09-11T09:16:42.223 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60307.log 2024-09-11T09:16:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38343.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38343.log.gz 2024-09-11T09:16:42.224 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50213.log 2024-09-11T09:16:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60307.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60307.log.gz 2024-09-11T09:16:42.225 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30696.log 2024-09-11T09:16:42.227 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50213.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50213.log.gz 2024-09-11T09:16:42.228 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30834.log 2024-09-11T09:16:42.229 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30696.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30696.log.gz 2024-09-11T09:16:42.229 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30163.log 2024-09-11T09:16:42.230 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.30834.log: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.30834.log.gz 2024-09-11T09:16:42.230 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59517.log 2024-09-11T09:16:42.231 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30163.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30163.log.gz 2024-09-11T09:16:42.231 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35575.log 2024-09-11T09:16:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59517.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59517.log.gz 2024-09-11T09:16:42.232 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53055.log 2024-09-11T09:16:42.239 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35575.log: 93.0% -- replaced with /var/log/ceph/cluster1-client.mirror.0.39791.log.gz 2024-09-11T09:16:42.248 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56519.log 2024-09-11T09:16:42.249 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53055.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53055.log.gz 2024-09-11T09:16:42.249 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60105.log 2024-09-11T09:16:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56519.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56519.log.gz 2024-09-11T09:16:42.252 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43407.log 2024-09-11T09:16:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60105.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60105.log.gz 2024-09-11T09:16:42.253 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30521.log 2024-09-11T09:16:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43407.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43407.log.gz 2024-09-11T09:16:42.254 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45830.log 2024-09-11T09:16:42.255 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30521.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30521.log.gz 2024-09-11T09:16:42.255 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45806.log 2024-09-11T09:16:42.256 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45830.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45830.log.gz 2024-09-11T09:16:42.257 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51261.log 2024-09-11T09:16:42.257 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45806.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45806.log.gz 2024-09-11T09:16:42.257 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55453.log 2024-09-11T09:16:42.259 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51261.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51261.log.gz 2024-09-11T09:16:42.259 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31511.log 2024-09-11T09:16:42.261 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55453.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55453.log.gz 2024-09-11T09:16:42.261 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57372.log 2024-09-11T09:16:42.262 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31511.log: 56.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31511.log.gz 2024-09-11T09:16:42.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51581.log 2024-09-11T09:16:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57372.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.57372.log.gz 2024-09-11T09:16:42.263 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59880.log 2024-09-11T09:16:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51581.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51581.log.gz 2024-09-11T09:16:42.265 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58546.log 2024-09-11T09:16:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59880.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.59880.log.gz 2024-09-11T09:16:42.266 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45714.log 2024-09-11T09:16:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58546.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58546.log.gz 2024-09-11T09:16:42.267 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55415.log 2024-09-11T09:16:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45714.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45714.log.gz 2024-09-11T09:16:42.269 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.30898.log 2024-09-11T09:16:42.270 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55415.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55415.log.gz 2024-09-11T09:16:42.271 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38992.log 2024-09-11T09:16:42.273 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.30898.log: 92.2% -- replaced with /var/log/ceph/cluster2-client.mirror.0.30898.log.gz 2024-09-11T09:16:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60442.log 2024-09-11T09:16:42.274 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38992.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38992.log.gz 2024-09-11T09:16:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33879.log 2024-09-11T09:16:42.275 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60442.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60442.log.gz 2024-09-11T09:16:42.276 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63892.log 2024-09-11T09:16:42.277 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33879.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.33879.log.gz 2024-09-11T09:16:42.277 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54852.log 2024-09-11T09:16:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63892.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63892.log.gz 2024-09-11T09:16:42.279 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65629.log 2024-09-11T09:16:42.280 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54852.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54852.log.gz 2024-09-11T09:16:42.280 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61522.log 2024-09-11T09:16:42.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65629.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65629.log.gz 2024-09-11T09:16:42.282 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64007.log 2024-09-11T09:16:42.282 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61522.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61522.log.gz 2024-09-11T09:16:42.283 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30317.log 2024-09-11T09:16:42.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64007.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64007.log.gz 2024-09-11T09:16:42.284 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66956.log 2024-09-11T09:16:42.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30317.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30317.log.gz 2024-09-11T09:16:42.285 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49051.log 2024-09-11T09:16:42.286 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.66956.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66956.log.gz 2024-09-11T09:16:42.286 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45626.log 2024-09-11T09:16:42.287 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49051.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49051.log.gz 2024-09-11T09:16:42.287 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42052.log 2024-09-11T09:16:42.289 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45626.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.45626.log.gz 2024-09-11T09:16:42.290 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41004.log 2024-09-11T09:16:42.290 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42052.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42052.log.gz 2024-09-11T09:16:42.291 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.36763.log 2024-09-11T09:16:42.292 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41004.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41004.log.gz 2024-09-11T09:16:42.292 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31221.log 2024-09-11T09:16:42.295 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.36763.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.36763.log.gz 2024-09-11T09:16:42.295 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39939.log 2024-09-11T09:16:42.296 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31221.log: 4.7% -- replaced with /var/log/ceph/cluster1-client.mirror.31221.log.gz 2024-09-11T09:16:42.296 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40811.log 2024-09-11T09:16:42.298 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39939.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65474.log.gz 2024-09-11T09:16:42.298 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35063.log 2024-09-11T09:16:42.302 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40811.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64284.log.gz 2024-09-11T09:16:42.304 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37879.log 2024-09-11T09:16:42.305 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35063.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35063.log.gz 2024-09-11T09:16:42.305 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43466.log 2024-09-11T09:16:42.311 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37879.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37879.log.gz 2024-09-11T09:16:42.311 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25422.log 2024-09-11T09:16:42.316 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43466.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42518.log.gz 2024-09-11T09:16:42.328 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45071.log 2024-09-11T09:16:42.329 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.25422.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25422.log.gz 2024-09-11T09:16:42.329 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39589.log 2024-09-11T09:16:42.330 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45071.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45071.log.gz 2024-09-11T09:16:42.330 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35191.log 2024-09-11T09:16:42.331 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39589.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39589.log.gz 2024-09-11T09:16:42.332 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35837.log 2024-09-11T09:16:42.340 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35191.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51760.log 2024-09-11T09:16:42.341 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35837.log: 88.9% -- replaced with /var/log/ceph/cluster1-client.mirror.35837.log.gz 2024-09-11T09:16:42.356 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38423.log 2024-09-11T09:16:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51760.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.51760.log.gz 2024-09-11T09:16:42.358 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.32324.log 2024-09-11T09:16:42.359 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38423.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38423.log.gz 2024-09-11T09:16:42.359 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50454.log 2024-09-11T09:16:42.365 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.32324.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.32324.log.gz 2024-09-11T09:16:42.371 INFO:teuthology.orchestra.run.smithi077.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40811.log.gz 2024-09-11T09:16:42.380 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53079.log 2024-09-11T09:16:42.382 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50454.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50454.log.gz 2024-09-11T09:16:42.383 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30857.log 2024-09-11T09:16:42.384 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53079.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.53079.log.gz 2024-09-11T09:16:42.384 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39337.log 2024-09-11T09:16:42.395 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.30857.log: 94.1%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40900.log 2024-09-11T09:16:42.397 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39337.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39337.log.gz 2024-09-11T09:16:42.397 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.30857.log.gz 2024-09-11T09:16:42.400 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36330.log 2024-09-11T09:16:42.401 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40900.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40900.log.gz 2024-09-11T09:16:42.401 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46047.log 2024-09-11T09:16:42.402 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.36330.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.admin.36330.log.gz 2024-09-11T09:16:42.402 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63963.log 2024-09-11T09:16:42.403 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46047.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46047.log.gz 2024-09-11T09:16:42.404 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/ceph.tmp-client.admin.21606.log 2024-09-11T09:16:42.405 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63963.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63963.log.gz 2024-09-11T09:16:42.405 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56773.log 2024-09-11T09:16:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/ceph.tmp-client.admin.21606.log: 0.0% -- replaced with /var/log/ceph/ceph.tmp-client.admin.21606.log.gz 2024-09-11T09:16:42.406 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.44431.log 2024-09-11T09:16:42.407 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56773.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56773.log.gz 2024-09-11T09:16:42.407 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25531.log 2024-09-11T09:16:42.424 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.1.44431.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66720.log 2024-09-11T09:16:42.425 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.25531.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25531.log.gz 2024-09-11T09:16:42.440 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48786.log 2024-09-11T09:16:42.441 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66720.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66720.log.gz 2024-09-11T09:16:42.441 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53508.log 2024-09-11T09:16:42.442 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48786.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48786.log.gz 2024-09-11T09:16:42.456 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46192.log 2024-09-11T09:16:42.457 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53508.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53508.log.gz 2024-09-11T09:16:42.472 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33158.log 2024-09-11T09:16:42.473 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46192.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46192.log.gz 2024-09-11T09:16:42.480 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49075.log 2024-09-11T09:16:42.481 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33158.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33158.log.gz 2024-09-11T09:16:42.486 INFO:teuthology.orchestra.run.smithi077.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.1.44431.log.gz 2024-09-11T09:16:42.486 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55099.log 2024-09-11T09:16:42.487 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49075.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49075.log.gz 2024-09-11T09:16:42.488 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58095.log 2024-09-11T09:16:42.489 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55099.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55099.log.gz 2024-09-11T09:16:42.489 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57909.log 2024-09-11T09:16:42.491 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58095.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58095.log.gz 2024-09-11T09:16:42.492 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44061.log 2024-09-11T09:16:42.494 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57909.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57909.log.gz 2024-09-11T09:16:42.495 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55951.log 2024-09-11T09:16:42.504 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44061.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35575.log.gz 2024-09-11T09:16:42.505 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45167.log 2024-09-11T09:16:42.506 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55951.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55951.log.gz 2024-09-11T09:16:42.506 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30079.log 2024-09-11T09:16:42.507 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45167.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.45167.log.gz 2024-09-11T09:16:42.507 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40247.log 2024-09-11T09:16:42.508 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30079.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30079.log.gz 2024-09-11T09:16:42.508 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55212.log 2024-09-11T09:16:42.509 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40247.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40247.log.gz 2024-09-11T09:16:42.510 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35399.log 2024-09-11T09:16:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55212.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55212.log.gz 2024-09-11T09:16:42.511 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56752.log 2024-09-11T09:16:42.512 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35399.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35399.log.gz 2024-09-11T09:16:42.513 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66852.log 2024-09-11T09:16:42.514 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56752.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56752.log.gz 2024-09-11T09:16:42.514 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46110.log 2024-09-11T09:16:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.66852.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66852.log.gz 2024-09-11T09:16:42.515 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38160.log 2024-09-11T09:16:42.516 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46110.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46110.log.gz 2024-09-11T09:16:42.517 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65244.log 2024-09-11T09:16:42.522 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38160.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38160.log.gz 2024-09-11T09:16:42.523 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44190.log 2024-09-11T09:16:42.524 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65244.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.65244.log.gz 2024-09-11T09:16:42.524 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.64903.log 2024-09-11T09:16:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44190.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44190.log.gz 2024-09-11T09:16:42.525 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51386.log 2024-09-11T09:16:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.64903.log: 92.7% -- replaced with /var/log/ceph/cluster1-client.mirror.0.64903.log.gz 2024-09-11T09:16:42.529 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50928.log 2024-09-11T09:16:42.530 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51386.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.51386.log.gz 2024-09-11T09:16:42.531 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31153.log 2024-09-11T09:16:42.533 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50928.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.50928.log.gz 2024-09-11T09:16:42.533 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58178.log 2024-09-11T09:16:42.534 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.31153.log: 79.9% -- replaced with /var/log/ceph/cluster1-client.admin.31153.log.gz 2024-09-11T09:16:42.534 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37535.log 2024-09-11T09:16:42.535 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58178.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58178.log.gz 2024-09-11T09:16:42.536 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37578.log 2024-09-11T09:16:42.541 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37535.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37535.log.gz 2024-09-11T09:16:42.542 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41046.log 2024-09-11T09:16:42.549 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37578.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37578.log.gz 2024-09-11T09:16:42.549 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.58262.log 2024-09-11T09:16:42.556 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41046.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35633.log 2024-09-11T09:16:42.558 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.2.58262.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43054.log.gz 2024-09-11T09:16:42.569 INFO:teuthology.orchestra.run.smithi077.stderr: 92.2%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42024.log 2024-09-11T09:16:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.35633.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.admin.35633.log.gz 2024-09-11T09:16:42.570 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.58262.log.gz 2024-09-11T09:16:42.570 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30165.log 2024-09-11T09:16:42.584 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42024.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44125.log 2024-09-11T09:16:42.585 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30165.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30165.log.gz 2024-09-11T09:16:42.595 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35191.log.gz 2024-09-11T09:16:42.595 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37515.log 2024-09-11T09:16:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44125.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44125.log.gz 2024-09-11T09:16:42.596 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50610.log 2024-09-11T09:16:42.602 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37515.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37515.log.gz 2024-09-11T09:16:42.603 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42476.log 2024-09-11T09:16:42.603 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50610.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50610.log.gz 2024-09-11T09:16:42.604 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58455.log 2024-09-11T09:16:42.605 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42476.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42476.log.gz 2024-09-11T09:16:42.605 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42555.log 2024-09-11T09:16:42.606 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58455.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58455.log.gz 2024-09-11T09:16:42.607 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37836.log 2024-09-11T09:16:42.608 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42555.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42555.log.gz 2024-09-11T09:16:42.609 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55120.log 2024-09-11T09:16:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37836.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37836.log.gz 2024-09-11T09:16:42.614 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59703.log 2024-09-11T09:16:42.617 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55120.log: 92.3% -- replaced with /var/log/ceph/cluster2-client.mirror.55120.log.gz 2024-09-11T09:16:42.617 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55682.log 2024-09-11T09:16:42.618 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59703.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59703.log.gz 2024-09-11T09:16:42.618 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52196.log 2024-09-11T09:16:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55682.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55682.log.gz 2024-09-11T09:16:42.620 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55545.log 2024-09-11T09:16:42.622 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52196.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52196.log.gz 2024-09-11T09:16:42.623 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43466.log.gz 2024-09-11T09:16:42.623 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31460.log 2024-09-11T09:16:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39069.log 2024-09-11T09:16:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55545.log: /var/log/ceph/cluster2-client.mirror.31460.log: 58.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31460.log.gz 2024-09-11T09:16:42.624 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55545.log.gz 2024-09-11T09:16:42.624 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64225.log 2024-09-11T09:16:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41066.log 2024-09-11T09:16:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39069.log: /var/log/ceph/cluster2-client.mirror.64225.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39069.log.gz 2024-09-11T09:16:42.625 INFO:teuthology.orchestra.run.smithi077.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64225.log.gz 2024-09-11T09:16:42.625 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46148.log 2024-09-11T09:16:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41066.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41066.log.gz 2024-09-11T09:16:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59042.log 2024-09-11T09:16:42.626 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43190.log 2024-09-11T09:16:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46148.log: /var/log/ceph/cluster2-client.mirror.59042.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59042.log.gz 2024-09-11T09:16:42.627 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46148.log.gz 2024-09-11T09:16:42.627 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40572.log 2024-09-11T09:16:42.629 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30242.log 2024-09-11T09:16:42.629 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43190.log: /var/log/ceph/cluster2-client.mirror.40572.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43190.log.gz 2024-09-11T09:16:42.629 INFO:teuthology.orchestra.run.smithi077.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40572.log.gz 2024-09-11T09:16:42.629 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58354.log 2024-09-11T09:16:42.630 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47558.log 2024-09-11T09:16:42.630 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30242.log: /var/log/ceph/cluster2-client.mirror.58354.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30242.log.gz 2024-09-11T09:16:42.630 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45893.log 2024-09-11T09:16:42.631 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47558.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47558.log.gz 2024-09-11T09:16:42.631 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.66497.log 2024-09-11T09:16:42.632 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58354.log.gz/var/log/ceph/cluster2-client.mirror.45893.log: 2024-09-11T09:16:42.632 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45893.log.gz 2024-09-11T09:16:42.633 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60352.log 2024-09-11T09:16:42.633 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38303.log 2024-09-11T09:16:42.634 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.66497.log: /var/log/ceph/cluster1-client.mirror.60352.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60352.log.gz 2024-09-11T09:16:42.634 INFO:teuthology.orchestra.run.smithi077.stderr: 91.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.66497.log.gz 2024-09-11T09:16:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52155.log 2024-09-11T09:16:42.635 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57641.log 2024-09-11T09:16:42.636 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38303.log: /var/log/ceph/cluster1-client.mirror.52155.log: 88.0% 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38303.log.gz 2024-09-11T09:16:42.636 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.52155.log.gz 2024-09-11T09:16:42.636 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59434.log 2024-09-11T09:16:42.637 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44593.log 2024-09-11T09:16:42.637 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57641.log: /var/log/ceph/cluster2-client.mirror.59434.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57641.log.gz 2024-09-11T09:16:42.639 INFO:teuthology.orchestra.run.smithi077.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59434.log.gz 2024-09-11T09:16:42.639 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48830.log 2024-09-11T09:16:42.639 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44593.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44822.log 2024-09-11T09:16:42.640 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.48830.log: -- replaced with /var/log/ceph/cluster1-client.mirror.44593.log.gz 2024-09-11T09:16:42.641 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48830.log.gz 2024-09-11T09:16:42.641 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53600.log 2024-09-11T09:16:42.642 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53576.log 2024-09-11T09:16:42.642 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44822.log: /var/log/ceph/cluster1-client.mirror.53600.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44822.log.gz 2024-09-11T09:16:42.642 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.53600.log.gz 2024-09-11T09:16:42.642 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35147.log 2024-09-11T09:16:42.643 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41103.log 2024-09-11T09:16:42.643 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53576.log: /var/log/ceph/cluster1-client.mirror.35147.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53576.log.gz 2024-09-11T09:16:42.643 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54568.log 2024-09-11T09:16:42.643 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41103.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41103.log.gz 2024-09-11T09:16:42.644 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65671.log 2024-09-11T09:16:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54568.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.54568.log.gz 2024-09-11T09:16:42.645 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59455.log 2024-09-11T09:16:42.646 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65671.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65671.log.gz 2024-09-11T09:16:42.646 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42975.log 2024-09-11T09:16:42.647 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59455.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59455.log.gz 2024-09-11T09:16:42.648 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31332.log 2024-09-11T09:16:42.648 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35147.log.gz 2024-09-11T09:16:42.652 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56814.log 2024-09-11T09:16:42.653 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.31332.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31332.log.gz 2024-09-11T09:16:42.653 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62136.log 2024-09-11T09:16:42.654 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56814.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56814.log.gz 2024-09-11T09:16:42.654 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38711.log 2024-09-11T09:16:42.655 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.62136.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62136.log.gz 2024-09-11T09:16:42.656 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44146.log 2024-09-11T09:16:42.657 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38711.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38711.log.gz 2024-09-11T09:16:42.657 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31651.log 2024-09-11T09:16:42.658 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44146.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44146.log.gz 2024-09-11T09:16:42.658 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47285.log 2024-09-11T09:16:42.660 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31651.log: 94.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31651.log.gz 2024-09-11T09:16:42.661 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36560.log 2024-09-11T09:16:42.662 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47285.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47285.log.gz 2024-09-11T09:16:42.662 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38903.log 2024-09-11T09:16:42.667 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36560.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39939.log.gz 2024-09-11T09:16:42.668 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.36560.log.gz 2024-09-11T09:16:42.668 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33677.log 2024-09-11T09:16:42.669 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43905.log 2024-09-11T09:16:42.670 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.38903.log: /var/log/ceph/cluster1-client.mirror.33677.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33677.log.gz 2024-09-11T09:16:42.670 INFO:teuthology.orchestra.run.smithi077.stderr: 93.7% -- replaced with /var/log/ceph/cluster1-client.mirror.38903.log.gz 2024-09-11T09:16:42.670 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66452.log 2024-09-11T09:16:42.670 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.41355.log 2024-09-11T09:16:42.671 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43905.log: /var/log/ceph/cluster2-client.mirror.66452.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.66452.log.gz 2024-09-11T09:16:42.678 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.43905.log.gz 2024-09-11T09:16:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40730.log 2024-09-11T09:16:42.679 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32742.log 2024-09-11T09:16:42.680 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.41355.log: /var/log/ceph/cluster2-client.mirror.40730.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40730.log.gz 2024-09-11T09:16:42.681 INFO:teuthology.orchestra.run.smithi077.stderr: 95.0% -- replaced with /var/log/ceph/cluster1-client.mirror.41355.log.gz 2024-09-11T09:16:42.682 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36688.log 2024-09-11T09:16:42.683 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57455.log 2024-09-11T09:16:42.683 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.32742.log: /var/log/ceph/cluster1-client.admin.36688.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.32742.log.gz 2024-09-11T09:16:42.683 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster1-client.admin.36688.log.gz 2024-09-11T09:16:42.684 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65778.log 2024-09-11T09:16:42.684 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57455.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57455.log.gz 2024-09-11T09:16:42.684 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40418.log 2024-09-11T09:16:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34285.log 2024-09-11T09:16:42.685 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65778.log: /var/log/ceph/cluster2-client.mirror.40418.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40418.log.gz 2024-09-11T09:16:42.686 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.65778.log.gz 2024-09-11T09:16:42.686 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34426.log 2024-09-11T09:16:42.686 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50634.log 2024-09-11T09:16:42.687 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34285.log: /var/log/ceph/cluster2-client.mirror.34426.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.34426.log.gz 2024-09-11T09:16:42.700 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45646.log 2024-09-11T09:16:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50634.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.50634.log.gz 2024-09-11T09:16:42.701 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33837.log 2024-09-11T09:16:42.703 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45646.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.45646.log.gz 2024-09-11T09:16:42.704 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59248.log 2024-09-11T09:16:42.709 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33837.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33837.log.gz 2024-09-11T09:16:42.710 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.42133.log 2024-09-11T09:16:42.712 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59248.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59248.log.gz 2024-09-11T09:16:42.713 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47064.log 2024-09-11T09:16:42.727 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.1.42133.log: 93.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64051.log 2024-09-11T09:16:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47064.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47064.log.gz 2024-09-11T09:16:42.728 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60510.log 2024-09-11T09:16:42.729 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64051.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64051.log.gz 2024-09-11T09:16:42.730 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.1.42133.log.gz 2024-09-11T09:16:42.730 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64604.log 2024-09-11T09:16:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60510.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60510.log.gz 2024-09-11T09:16:42.731 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52501.log 2024-09-11T09:16:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64604.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64604.log.gz 2024-09-11T09:16:42.732 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48726.log 2024-09-11T09:16:42.733 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52501.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52501.log.gz 2024-09-11T09:16:42.733 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60690.log 2024-09-11T09:16:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48726.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48726.log.gz 2024-09-11T09:16:42.736 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60240.log 2024-09-11T09:16:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60690.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60690.log.gz 2024-09-11T09:16:42.737 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60577.log 2024-09-11T09:16:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60240.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60240.log.gz 2024-09-11T09:16:42.738 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47044.log 2024-09-11T09:16:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60577.log: 86.6% -- replaced with /var/log/ceph/cluster1-client.mirror.60577.log.gz 2024-09-11T09:16:42.739 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38180.log 2024-09-11T09:16:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47044.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47044.log.gz 2024-09-11T09:16:42.741 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41854.log 2024-09-11T09:16:42.746 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.38180.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.38180.log.gz 2024-09-11T09:16:42.747 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55590.log 2024-09-11T09:16:42.760 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41854.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40987.log 2024-09-11T09:16:42.762 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55590.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.55590.log.gz 2024-09-11T09:16:42.763 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63986.log 2024-09-11T09:16:42.763 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40987.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40987.log.gz 2024-09-11T09:16:42.776 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32646.log 2024-09-11T09:16:42.777 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63986.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.63986.log.gz 2024-09-11T09:16:42.780 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65123.log 2024-09-11T09:16:42.781 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32646.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.32646.log.gz 2024-09-11T09:16:42.796 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52728.log 2024-09-11T09:16:42.797 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65123.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65123.log.gz 2024-09-11T09:16:42.804 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54739.log 2024-09-11T09:16:42.812 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52728.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53027.log 2024-09-11T09:16:42.813 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54739.log: 94.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54739.log.gz 2024-09-11T09:16:42.821 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41716.log 2024-09-11T09:16:42.822 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53027.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.53027.log.gz 2024-09-11T09:16:42.828 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43153.log 2024-09-11T09:16:42.829 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41716.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41716.log.gz 2024-09-11T09:16:42.835 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35511.log 2024-09-11T09:16:42.836 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43153.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43153.log.gz 2024-09-11T09:16:42.836 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.44061.log.gz 2024-09-11T09:16:42.836 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.52728.log.gz 2024-09-11T09:16:42.836 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48319.log 2024-09-11T09:16:42.842 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35511.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35511.log.gz 2024-09-11T09:16:42.842 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61342.log 2024-09-11T09:16:42.843 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48319.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48319.log.gz 2024-09-11T09:16:42.843 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44083.log 2024-09-11T09:16:42.844 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61342.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61342.log.gz 2024-09-11T09:16:42.844 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47942.log 2024-09-11T09:16:42.844 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44083.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44083.log.gz 2024-09-11T09:16:42.845 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46327.log 2024-09-11T09:16:42.846 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47942.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47942.log.gz 2024-09-11T09:16:42.846 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34522.log 2024-09-11T09:16:42.847 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46327.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46327.log.gz 2024-09-11T09:16:42.847 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.38775.log 2024-09-11T09:16:42.848 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34522.log: 88.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34522.log.gz 2024-09-11T09:16:42.849 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61995.log 2024-09-11T09:16:42.856 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.38775.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59331.log 2024-09-11T09:16:42.857 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61995.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61995.log.gz 2024-09-11T09:16:42.857 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.38775.log.gz 2024-09-11T09:16:42.857 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67162.log 2024-09-11T09:16:42.858 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59331.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59331.log.gz 2024-09-11T09:16:42.859 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65267.log 2024-09-11T09:16:42.859 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.67162.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67162.log.gz 2024-09-11T09:16:42.859 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32626.log 2024-09-11T09:16:42.860 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65267.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65267.log.gz 2024-09-11T09:16:42.860 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45581.log 2024-09-11T09:16:42.862 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32626.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32626.log.gz 2024-09-11T09:16:42.862 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30746.log 2024-09-11T09:16:42.862 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45581.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45581.log.gz 2024-09-11T09:16:42.863 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34157.log 2024-09-11T09:16:42.863 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30746.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30746.log.gz 2024-09-11T09:16:42.864 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66366.log 2024-09-11T09:16:42.864 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34157.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34157.log.gz 2024-09-11T09:16:42.865 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59620.log 2024-09-11T09:16:42.866 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66366.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.66366.log.gz 2024-09-11T09:16:42.866 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51108.log 2024-09-11T09:16:42.868 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59620.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59620.log.gz 2024-09-11T09:16:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55569.log 2024-09-11T09:16:42.869 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51108.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51108.log.gz 2024-09-11T09:16:42.870 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63140.log 2024-09-11T09:16:42.871 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55569.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55569.log.gz 2024-09-11T09:16:42.871 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56402.log 2024-09-11T09:16:42.872 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63140.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63140.log.gz 2024-09-11T09:16:42.872 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50385.log 2024-09-11T09:16:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56402.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.56402.log.gz 2024-09-11T09:16:42.873 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55280.log 2024-09-11T09:16:42.874 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50385.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41046.log.gz 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50385.log.gz 2024-09-11T09:16:42.874 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:42.874 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60375.log 2024-09-11T09:16:42.876 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55280.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37644.log 2024-09-11T09:16:42.876 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60375.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55280.log.gz 2024-09-11T09:16:42.876 INFO:teuthology.orchestra.run.smithi077.stderr: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60375.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43523.log 2024-09-11T09:16:42.876 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:42.881 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37644.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37644.log.gz 2024-09-11T09:16:42.881 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60780.log 2024-09-11T09:16:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43523.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42915.log 2024-09-11T09:16:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60780.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43523.log.gz 2024-09-11T09:16:42.883 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60780.log.gz 2024-09-11T09:16:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.36715.log 2024-09-11T09:16:42.883 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42915.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47582.log 2024-09-11T09:16:42.883 INFO:teuthology.orchestra.run.smithi077.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42915.log.gz 2024-09-11T09:16:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.36715.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60960.log 2024-09-11T09:16:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47582.log: 88.0% -- replaced with /var/log/ceph/cluster1-client.admin.36715.log.gz 2024-09-11T09:16:42.885 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47582.log.gz 2024-09-11T09:16:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.54739.log 2024-09-11T09:16:42.885 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60960.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60960.log.gz 2024-09-11T09:16:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41227.log 2024-09-11T09:16:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33922.log 2024-09-11T09:16:42.886 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.54739.log: /var/log/ceph/cluster2-client.mirror.41227.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41227.log.gz 2024-09-11T09:16:42.887 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53444.log 2024-09-11T09:16:42.887 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42975.log.gz 2024-09-11T09:16:42.887 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.33922.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.33922.log.gz 2024-09-11T09:16:42.888 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56248.log 2024-09-11T09:16:42.888 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58477.log 2024-09-11T09:16:42.889 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53444.log: /var/log/ceph/cluster1-client.mirror.56248.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.56248.log.gz 2024-09-11T09:16:42.890 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57000.log 2024-09-11T09:16:42.890 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53444.log.gz 2024-09-11T09:16:42.890 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58477.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58477.log.gz 2024-09-11T09:16:42.891 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.33522.log 2024-09-11T09:16:42.892 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8%/var/log/ceph/cluster2-client.mirror.57000.log: -- replaced with /var/log/ceph/cluster2-client.mirror.42024.log.gz 2024-09-11T09:16:42.892 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50409.log 2024-09-11T09:16:42.892 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.33522.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57000.log.gz 2024-09-11T09:16:42.892 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54876.log 2024-09-11T09:16:42.893 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50409.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43133.log 2024-09-11T09:16:42.893 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54876.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50409.log.gz 2024-09-11T09:16:42.893 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44167.log 2024-09-11T09:16:42.893 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54876.log.gz 2024-09-11T09:16:42.895 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43133.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.33522.log.gz 2024-09-11T09:16:42.896 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51016.log 2024-09-11T09:16:42.896 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65374.log 2024-09-11T09:16:42.897 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44167.log: /var/log/ceph/cluster2-client.mirror.51016.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.44167.log.gz 2024-09-11T09:16:42.897 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42417.log 2024-09-11T09:16:42.897 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65374.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.51016.log.gz 2024-09-11T09:16:42.898 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46618.log 2024-09-11T09:16:42.898 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.65374.log.gz 2024-09-11T09:16:42.898 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33794.log 2024-09-11T09:16:42.900 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42417.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.0.54739.log.gz/var/log/ceph/cluster2-client.mirror.46618.log: 2024-09-11T09:16:42.900 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42417.log.gz 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46618.log.gz 2024-09-11T09:16:42.900 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:42.900 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44320.log 2024-09-11T09:16:42.900 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32829.log 2024-09-11T09:16:42.900 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33794.log: /var/log/ceph/cluster1-client.mirror.44320.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42222.log 2024-09-11T09:16:42.901 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44320.log.gz 2024-09-11T09:16:42.902 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32829.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44686.log 2024-09-11T09:16:42.902 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.32829.log.gz 2024-09-11T09:16:42.902 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42222.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42222.log.gz 2024-09-11T09:16:42.902 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51718.log 2024-09-11T09:16:42.902 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44686.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36667.log 2024-09-11T09:16:42.904 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51718.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.44686.log.gz 2024-09-11T09:16:42.904 INFO:teuthology.orchestra.run.smithi077.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51718.log.gz 2024-09-11T09:16:42.904 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37773.log 2024-09-11T09:16:42.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36667.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55393.log 2024-09-11T09:16:42.905 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.33794.log.gz 2024-09-11T09:16:42.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37773.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53936.log 2024-09-11T09:16:42.907 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55393.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55393.log.gz 2024-09-11T09:16:42.907 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48004.log 2024-09-11T09:16:42.908 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53936.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53936.log.gz 2024-09-11T09:16:42.909 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.42133.log 2024-09-11T09:16:42.910 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48004.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48004.log.gz 2024-09-11T09:16:42.910 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31375.log 2024-09-11T09:16:42.910 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37773.log.gz 2024-09-11T09:16:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.42133.log: 94.7% -- replaced with /var/log/ceph/cluster1-client.mirror.42133.log.gz 2024-09-11T09:16:42.911 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32531.log 2024-09-11T09:16:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.31375.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52083.log 2024-09-11T09:16:42.912 INFO:teuthology.orchestra.run.smithi077.stderr: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.31375.log.gz 2024-09-11T09:16:42.912 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32531.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63119.log 2024-09-11T09:16:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52083.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32531.log.gz 2024-09-11T09:16:42.913 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.52083.log.gz 2024-09-11T09:16:42.913 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41444.log 2024-09-11T09:16:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53801.log 2024-09-11T09:16:42.914 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63119.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.63119.log.gz/var/log/ceph/cluster2-client.mirror.41444.log: 2024-09-11T09:16:42.915 INFO:teuthology.orchestra.run.smithi077.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41444.log.gz 2024-09-11T09:16:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48497.log 2024-09-11T09:16:42.915 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53801.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48293.log 2024-09-11T09:16:42.915 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4%/var/log/ceph/cluster1-client.mirror.48497.log: -- replaced with /var/log/ceph/cluster1-client.mirror.53801.log.gz 2024-09-11T09:16:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52913.log 2024-09-11T09:16:42.916 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48293.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48293.log.gz 2024-09-11T09:16:42.917 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48497.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47651.log 2024-09-11T09:16:42.917 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:42.917 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52913.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.52913.log.gz 2024-09-11T09:16:42.918 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32880.log 2024-09-11T09:16:42.918 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47651.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36580.log 2024-09-11T09:16:42.919 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32880.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32880.log.gz 2024-09-11T09:16:42.919 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31443.log 2024-09-11T09:16:42.920 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36580.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47651.log.gz 2024-09-11T09:16:42.920 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44870.log 2024-09-11T09:16:42.921 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31443.log: 3.5% -- replaced with /var/log/ceph/cluster2-client.mirror.31443.log.gz 2024-09-11T09:16:42.921 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61905.log 2024-09-11T09:16:42.922 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44870.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44870.log.gz 2024-09-11T09:16:42.922 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38223.log 2024-09-11T09:16:42.923 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61905.log: 85.2% -- replaced with /var/log/ceph/cluster1-client.mirror.61905.log.gz 2024-09-11T09:16:42.924 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46847.log 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38223.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38223.log.gz 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36580.log.gz 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr: 90.1% -- replaced with /var/log/ceph/cluster2-mon.a.log.gz 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66620.log 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46847.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41876.log 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7%/var/log/ceph/cluster1-client.mirror.66620.log: -- replaced with /var/log/ceph/cluster2-client.mirror.46847.log.gz 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65974.log 2024-09-11T09:16:42.926 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66620.log.gz 2024-09-11T09:16:42.927 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41876.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33182.log 2024-09-11T09:16:42.928 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37189.log 2024-09-11T09:16:42.928 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65974.log: /var/log/ceph/cluster1-client.mirror.33182.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.65974.log.gz 2024-09-11T09:16:42.928 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.33182.log.gz 2024-09-11T09:16:42.928 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46216.log 2024-09-11T09:16:42.929 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38523.log 2024-09-11T09:16:42.929 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37189.log: /var/log/ceph/cluster1-client.mirror.46216.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46216.log.gz 2024-09-11T09:16:42.932 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38503.log 2024-09-11T09:16:42.934 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38523.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.38523.log.gz 2024-09-11T09:16:42.934 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.37189.log.gz 2024-09-11T09:16:42.935 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.31088.log 2024-09-11T09:16:42.935 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38503.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38008.log 2024-09-11T09:16:42.935 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.31088.log: 88.1% 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.31088.log.gz 2024-09-11T09:16:42.935 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38503.log.gz 2024-09-11T09:16:42.935 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46714.log 2024-09-11T09:16:42.936 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61252.log 2024-09-11T09:16:42.936 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.38008.log: /var/log/ceph/cluster1-client.mirror.46714.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46714.log.gz 2024-09-11T09:16:42.940 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38008.log.gz 2024-09-11T09:16:42.941 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36515.log 2024-09-11T09:16:42.941 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53757.log 2024-09-11T09:16:42.941 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61252.log: /var/log/ceph/cluster1-client.mirror.36515.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61252.log.gz 2024-09-11T09:16:42.942 INFO:teuthology.orchestra.run.smithi077.stderr: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36515.log.gz 2024-09-11T09:16:42.942 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32558.log 2024-09-11T09:16:42.943 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53757.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53757.log.gz 2024-09-11T09:16:42.943 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60892.log 2024-09-11T09:16:42.944 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51854.log 2024-09-11T09:16:42.945 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32558.log: /var/log/ceph/cluster1-client.mirror.60892.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60892.log.gz 2024-09-11T09:16:42.945 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.32558.log.gz 2024-09-11T09:16:42.945 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36356.log 2024-09-11T09:16:42.946 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35104.log 2024-09-11T09:16:42.946 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51854.log: /var/log/ceph/cluster2-client.mirror.36356.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51854.log.gz 2024-09-11T09:16:42.946 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56606.log 2024-09-11T09:16:42.947 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35104.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.36356.log.gz 2024-09-11T09:16:42.951 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35104.log.gz 2024-09-11T09:16:42.952 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59770.log 2024-09-11T09:16:42.952 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32933.log 2024-09-11T09:16:42.953 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56606.log: /var/log/ceph/cluster1-client.mirror.59770.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59770.log.gz 2024-09-11T09:16:42.953 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41854.log.gz 2024-09-11T09:16:42.954 INFO:teuthology.orchestra.run.smithi077.stderr: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56606.log.gz 2024-09-11T09:16:42.954 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51177.log 2024-09-11T09:16:42.955 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59791.log 2024-09-11T09:16:42.956 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32933.log: /var/log/ceph/cluster2-client.mirror.51177.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37816.log 2024-09-11T09:16:42.956 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59791.log: 1.3% -- replaced with /var/log/ceph/cluster1-client.mirror.59791.log.gz 2024-09-11T09:16:42.956 INFO:teuthology.orchestra.run.smithi077.stderr: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.32933.log.gz 2024-09-11T09:16:42.957 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55001.log 2024-09-11T09:16:42.957 INFO:teuthology.orchestra.run.smithi077.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.51177.log.gz 2024-09-11T09:16:42.957 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39406.log 2024-09-11T09:16:42.957 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37816.log: /var/log/ceph/cluster2-client.mirror.55001.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43715.log 2024-09-11T09:16:42.958 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39406.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.55001.log.gz 2024-09-11T09:16:42.959 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39406.log.gz 2024-09-11T09:16:42.962 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37816.log.gz 2024-09-11T09:16:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55632.log 2024-09-11T09:16:42.963 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64822.log 2024-09-11T09:16:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43715.log: /var/log/ceph/cluster2-client.mirror.55632.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38323.log 2024-09-11T09:16:42.964 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64822.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64822.log.gz 2024-09-11T09:16:42.964 INFO:teuthology.orchestra.run.smithi077.stderr: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.55632.log.gz 2024-09-11T09:16:42.972 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59744.log 2024-09-11T09:16:42.972 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52865.log 2024-09-11T09:16:42.973 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38323.log: /var/log/ceph/cluster2-client.mirror.59744.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38323.log.gz 2024-09-11T09:16:42.973 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57661.log 2024-09-11T09:16:42.974 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52865.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.52865.log.gz 2024-09-11T09:16:42.975 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47900.log 2024-09-11T09:16:42.975 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59744.log.gz 2024-09-11T09:16:42.977 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57661.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57661.log.gz 2024-09-11T09:16:42.977 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66645.log 2024-09-11T09:16:42.978 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40073.log 2024-09-11T09:16:42.978 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47900.log: /var/log/ceph/cluster1-client.mirror.66645.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.mirror.66645.log.gz 2024-09-11T09:16:42.979 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.47900.log.gz 2024-09-11T09:16:42.980 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65309.log 2024-09-11T09:16:42.980 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38775.log 2024-09-11T09:16:42.981 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40073.log: /var/log/ceph/cluster1-client.mirror.65309.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40073.log.gz 2024-09-11T09:16:42.981 INFO:teuthology.orchestra.run.smithi077.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65309.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47671.log 2024-09-11T09:16:42.981 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:42.985 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38775.log: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38775.log.gz 2024-09-11T09:16:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54143.log 2024-09-11T09:16:42.986 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47671.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65037.log 2024-09-11T09:16:42.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54143.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.47671.log.gz 2024-09-11T09:16:42.988 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54143.log.gz 2024-09-11T09:16:42.988 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55497.log 2024-09-11T09:16:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65037.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.65037.log.gz 2024-09-11T09:16:42.989 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48187.log 2024-09-11T09:16:42.990 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55497.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56669.log 2024-09-11T09:16:42.990 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6%/var/log/ceph/cluster2-client.mirror.48187.log: -- replaced with /var/log/ceph/cluster1-client.mirror.55497.log.gz 2024-09-11T09:16:42.991 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41523.log 2024-09-11T09:16:42.991 INFO:teuthology.orchestra.run.smithi077.stderr: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.48187.log.gz 2024-09-11T09:16:42.993 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56669.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.56669.log.gz 2024-09-11T09:16:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37297.log 2024-09-11T09:16:42.994 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41523.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44641.log 2024-09-11T09:16:42.994 INFO:teuthology.orchestra.run.smithi077.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41523.log.gz 2024-09-11T09:16:42.995 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37297.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32718.log 2024-09-11T09:16:42.995 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37297.log.gz/var/log/ceph/cluster1-client.mirror.44641.log: 2024-09-11T09:16:42.995 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49859.log 2024-09-11T09:16:42.995 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.44641.log.gz 2024-09-11T09:16:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.32718.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.32718.log.gz 2024-09-11T09:16:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60735.log 2024-09-11T09:16:42.996 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46369.log 2024-09-11T09:16:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49859.log: /var/log/ceph/cluster1-client.mirror.60735.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60735.log.gz 2024-09-11T09:16:42.997 INFO:teuthology.orchestra.run.smithi077.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.49859.log.gz 2024-09-11T09:16:42.997 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46755.log 2024-09-11T09:16:42.998 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51967.log 2024-09-11T09:16:42.999 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46369.log: /var/log/ceph/cluster1-client.mirror.46755.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.46369.log.gz 2024-09-11T09:16:42.999 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46964.log 2024-09-11T09:16:43.000 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51967.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46755.log.gz 2024-09-11T09:16:43.001 INFO:teuthology.orchestra.run.smithi077.stderr: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51967.log.gz 2024-09-11T09:16:43.001 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56938.log 2024-09-11T09:16:43.002 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34991.log 2024-09-11T09:16:43.002 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46964.log: /var/log/ceph/cluster2-client.mirror.56938.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56938.log.gz 2024-09-11T09:16:43.003 INFO:teuthology.orchestra.run.smithi077.stderr: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.46964.log.gz 2024-09-11T09:16:43.004 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60847.log 2024-09-11T09:16:43.004 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46871.log 2024-09-11T09:16:43.005 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34991.log: /var/log/ceph/cluster1-client.mirror.60847.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.34991.log.gz 2024-09-11T09:16:43.005 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60847.log.gz 2024-09-11T09:16:43.006 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55702.log 2024-09-11T09:16:43.006 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46871.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37386.log 2024-09-11T09:16:43.007 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.55702.log: -- replaced with /var/log/ceph/cluster1-client.mirror.46871.log.gz 2024-09-11T09:16:43.007 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55702.log.gz 2024-09-11T09:16:43.007 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41503.log 2024-09-11T09:16:43.007 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45557.log 2024-09-11T09:16:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37386.log: /var/log/ceph/cluster2-client.mirror.41503.log: 93.4% -- replaced with /var/log/ceph/cluster2-client.mirror.37386.log.gz 2024-09-11T09:16:43.012 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49734.log 2024-09-11T09:16:43.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45557.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.45557.log.gz 2024-09-11T09:16:43.013 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34178.log 2024-09-11T09:16:43.014 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49734.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49734.log.gz 2024-09-11T09:16:43.014 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45668.log 2024-09-11T09:16:43.020 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34178.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.34178.log.gz 2024-09-11T09:16:43.021 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39300.log 2024-09-11T09:16:43.021 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45668.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.45668.log.gz 2024-09-11T09:16:43.022 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.30857.log 2024-09-11T09:16:43.023 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39300.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39300.log.gz 2024-09-11T09:16:43.023 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43091.log 2024-09-11T09:16:43.036 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.3.30857.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48519.log 2024-09-11T09:16:43.037 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43091.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43091.log.gz 2024-09-11T09:16:43.048 INFO:teuthology.orchestra.run.smithi077.stderr: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.41876.log.gz 2024-09-11T09:16:43.052 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33206.log 2024-09-11T09:16:43.052 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64760.log 2024-09-11T09:16:43.053 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48519.log: /var/log/ceph/cluster1-client.mirror.33206.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.48519.log.gz 2024-09-11T09:16:43.053 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.33206.log.gz 2024-09-11T09:16:43.054 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58732.log 2024-09-11T09:16:43.055 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64760.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64760.log.gz 2024-09-11T09:16:43.055 INFO:teuthology.orchestra.run.smithi077.stderr: 93.3% -- replaced with /var/log/ceph/cluster1-client.mirror.3.30857.log.gz 2024-09-11T09:16:43.055 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36039.log 2024-09-11T09:16:43.056 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58732.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58732.log.gz 2024-09-11T09:16:43.056 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50337.log 2024-09-11T09:16:43.057 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42575.log 2024-09-11T09:16:43.057 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36039.log: /var/log/ceph/cluster1-client.mirror.50337.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50337.log.gz 2024-09-11T09:16:43.057 INFO:teuthology.orchestra.run.smithi077.stderr: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.36039.log.gz 2024-09-11T09:16:43.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33949.log 2024-09-11T09:16:43.058 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65953.log 2024-09-11T09:16:43.059 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42575.log: /var/log/ceph/cluster1-client.admin.33949.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42575.log.gz 2024-09-11T09:16:43.059 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.33949.log.gz 2024-09-11T09:16:43.059 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46984.log 2024-09-11T09:16:43.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49799.log 2024-09-11T09:16:43.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65953.log: /var/log/ceph/cluster1-client.mirror.46984.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.65953.log.gz 2024-09-11T09:16:43.060 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34546.log 2024-09-11T09:16:43.062 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49799.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46984.log.gz 2024-09-11T09:16:43.062 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49799.log.gz 2024-09-11T09:16:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41832.log 2024-09-11T09:16:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48923.log 2024-09-11T09:16:43.063 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34546.log: /var/log/ceph/cluster2-client.mirror.41832.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.34546.log.gz 2024-09-11T09:16:43.064 INFO:teuthology.orchestra.run.smithi077.stderr: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41832.log.gz 2024-09-11T09:16:43.064 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60015.log 2024-09-11T09:16:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48923.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48923.log.gz 2024-09-11T09:16:43.066 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56539.log 2024-09-11T09:16:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45230.log 2024-09-11T09:16:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60015.log: /var/log/ceph/cluster1-client.mirror.56539.log: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.60015.log.gz 2024-09-11T09:16:43.067 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43503.log 2024-09-11T09:16:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45230.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.45230.log.gz 2024-09-11T09:16:43.069 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56539.log.gz 2024-09-11T09:16:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59496.log 2024-09-11T09:16:43.069 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64305.log 2024-09-11T09:16:43.070 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43503.log: /var/log/ceph/cluster2-client.mirror.59496.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43503.log.gz 2024-09-11T09:16:43.070 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64567.log 2024-09-11T09:16:43.071 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64305.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.64305.log.gz 2024-09-11T09:16:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51926.log 2024-09-11T09:16:43.072 INFO:teuthology.orchestra.run.smithi077.stderr: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.59496.log.gz 2024-09-11T09:16:43.072 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64567.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64567.log.gz 2024-09-11T09:16:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61815.log 2024-09-11T09:16:43.073 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66765.log 2024-09-11T09:16:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51926.log: /var/log/ceph/cluster1-client.mirror.61815.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61815.log.gz 2024-09-11T09:16:43.074 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51926.log.gz 2024-09-11T09:16:43.074 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60465.log 2024-09-11T09:16:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55521.log 2024-09-11T09:16:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.66765.log: /var/log/ceph/cluster1-client.mirror.60465.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66765.log.gz 2024-09-11T09:16:43.075 INFO:teuthology.orchestra.run.smithi077.stderr: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.60465.log.gz 2024-09-11T09:16:43.075 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60150.log 2024-09-11T09:16:43.076 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55521.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.55521.log.gz 2024-09-11T09:16:43.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40505.log 2024-09-11T09:16:43.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34812.log 2024-09-11T09:16:43.077 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60150.log: /var/log/ceph/cluster2-client.mirror.40505.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60150.log.gz 2024-09-11T09:16:43.077 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35423.log 2024-09-11T09:16:43.078 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34812.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40505.log.gz 2024-09-11T09:16:43.084 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34812.log.gz 2024-09-11T09:16:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53127.log 2024-09-11T09:16:43.085 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55999.log 2024-09-11T09:16:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35423.log: /var/log/ceph/cluster1-client.mirror.53127.log: 88.8% -- replaced with /var/log/ceph/cluster1-client.mirror.35423.log.gz 2024-09-11T09:16:43.086 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53127.log.gz 2024-09-11T09:16:43.086 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36623.log 2024-09-11T09:16:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64626.log 2024-09-11T09:16:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55999.log: /var/log/ceph/cluster1-client.mirror.36623.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55999.log.gz 2024-09-11T09:16:43.087 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61140.log 2024-09-11T09:16:43.092 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64626.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.36623.log.gz 2024-09-11T09:16:43.104 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57703.log 2024-09-11T09:16:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61140.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61140.log.gz 2024-09-11T09:16:43.105 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51625.log 2024-09-11T09:16:43.106 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57703.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57703.log.gz 2024-09-11T09:16:43.107 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40589.log 2024-09-11T09:16:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51625.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51625.log.gz 2024-09-11T09:16:43.108 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40440.log 2024-09-11T09:16:43.109 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40589.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40589.log.gz 2024-09-11T09:16:43.110 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35933.log 2024-09-11T09:16:43.115 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40440.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36667.log.gz 2024-09-11T09:16:43.116 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61612.log 2024-09-11T09:16:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35933.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.35933.log.gz 2024-09-11T09:16:43.117 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30105.log 2024-09-11T09:16:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61612.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61612.log.gz 2024-09-11T09:16:43.118 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30671.log 2024-09-11T09:16:43.119 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30105.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30105.log.gz 2024-09-11T09:16:43.119 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47160.log 2024-09-11T09:16:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30671.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30671.log.gz 2024-09-11T09:16:43.120 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35885.log 2024-09-11T09:16:43.121 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47160.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47160.log.gz 2024-09-11T09:16:43.121 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56711.log 2024-09-11T09:16:43.122 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35885.log: 89.1% -- replaced with /var/log/ceph/cluster1-client.mirror.35885.log.gz 2024-09-11T09:16:43.123 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58773.log 2024-09-11T09:16:43.123 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56711.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56711.log.gz 2024-09-11T09:16:43.124 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34674.log 2024-09-11T09:16:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58773.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58773.log.gz 2024-09-11T09:16:43.125 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36467.log 2024-09-11T09:16:43.140 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34674.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58013.log 2024-09-11T09:16:43.141 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36467.log: 89.2% -- replaced with /var/log/ceph/cluster1-client.mirror.36467.log.gz 2024-09-11T09:16:43.141 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52573.log 2024-09-11T09:16:43.142 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58013.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.58013.log.gz 2024-09-11T09:16:43.156 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44039.log 2024-09-11T09:16:43.157 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52573.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.52573.log.gz 2024-09-11T09:16:43.157 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56897.log 2024-09-11T09:16:43.158 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44039.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44039.log.gz 2024-09-11T09:16:43.159 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61972.log 2024-09-11T09:16:43.160 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56897.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56897.log.gz 2024-09-11T09:16:43.164 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65016.log 2024-09-11T09:16:43.176 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61972.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61972.log.gz 2024-09-11T09:16:43.180 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51084.log 2024-09-11T09:16:43.181 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65016.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65016.log.gz 2024-09-11T09:16:43.196 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66345.log 2024-09-11T09:16:43.197 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51084.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51084.log.gz 2024-09-11T09:16:43.207 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34285.log.gz 2024-09-11T09:16:43.208 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54327.log 2024-09-11T09:16:43.209 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.66345.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.66345.log.gz 2024-09-11T09:16:43.209 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58628.log 2024-09-11T09:16:43.210 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54327.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.54327.log.gz 2024-09-11T09:16:43.210 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40166.log 2024-09-11T09:16:43.213 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58628.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.58628.log.gz 2024-09-11T09:16:43.213 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44345.log 2024-09-11T09:16:43.214 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40166.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40166.log.gz 2024-09-11T09:16:43.214 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40609.log 2024-09-11T09:16:43.215 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44345.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.44345.log.gz 2024-09-11T09:16:43.216 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40789.log 2024-09-11T09:16:43.217 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40609.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40609.log.gz 2024-09-11T09:16:43.217 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43656.log 2024-09-11T09:16:43.228 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40789.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41207.log 2024-09-11T09:16:43.229 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43656.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43656.log.gz 2024-09-11T09:16:43.241 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59393.log 2024-09-11T09:16:43.241 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43133.log.gz 2024-09-11T09:16:43.241 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41207.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41207.log.gz 2024-09-11T09:16:43.242 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34061.log 2024-09-11T09:16:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59393.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.59393.log.gz 2024-09-11T09:16:43.243 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48943.log 2024-09-11T09:16:43.244 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34061.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.34061.log.gz 2024-09-11T09:16:43.244 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56649.log 2024-09-11T09:16:43.247 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48943.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.48943.log.gz 2024-09-11T09:16:43.247 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56289.log 2024-09-11T09:16:43.248 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56649.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56649.log.gz 2024-09-11T09:16:43.248 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34587.log 2024-09-11T09:16:43.250 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56289.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56289.log.gz 2024-09-11T09:16:43.251 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36399.log 2024-09-11T09:16:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34587.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34587.log.gz 2024-09-11T09:16:43.257 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56627.log 2024-09-11T09:16:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36399.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36399.log.gz 2024-09-11T09:16:43.258 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39234.log 2024-09-11T09:16:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56627.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.56627.log.gz 2024-09-11T09:16:43.260 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53331.log 2024-09-11T09:16:43.261 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39234.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39234.log.gz 2024-09-11T09:16:43.261 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65394.log 2024-09-11T09:16:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53331.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53331.log.gz 2024-09-11T09:16:43.262 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61882.log 2024-09-11T09:16:43.265 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65394.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65394.log.gz 2024-09-11T09:16:43.265 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53711.log 2024-09-11T09:16:43.266 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61882.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61882.log.gz 2024-09-11T09:16:43.266 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34263.log 2024-09-11T09:16:43.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53711.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.53711.log.gz 2024-09-11T09:16:43.268 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56358.log 2024-09-11T09:16:43.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34263.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34263.log.gz 2024-09-11T09:16:43.270 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46168.log 2024-09-11T09:16:43.271 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56358.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.56358.log.gz 2024-09-11T09:16:43.271 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50474.log 2024-09-11T09:16:43.272 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46168.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46168.log.gz 2024-09-11T09:16:43.273 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59414.log 2024-09-11T09:16:43.275 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50474.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50474.log.gz 2024-09-11T09:16:43.275 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59641.log 2024-09-11T09:16:43.276 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59414.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59414.log.gz 2024-09-11T09:16:43.277 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38263.log 2024-09-11T09:16:43.277 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59641.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59641.log.gz 2024-09-11T09:16:43.278 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47136.log 2024-09-11T09:16:43.279 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38263.log: 88.7% -- replaced with /var/log/ceph/cluster2-client.mirror.38263.log.gz 2024-09-11T09:16:43.279 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43994.log 2024-09-11T09:16:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47136.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47136.log.gz 2024-09-11T09:16:43.280 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25398.log 2024-09-11T09:16:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43994.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43994.log.gz 2024-09-11T09:16:43.281 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49996.log 2024-09-11T09:16:43.283 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.25398.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25398.log.gz 2024-09-11T09:16:43.283 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43715.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52889.log 2024-09-11T09:16:43.283 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:43.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49996.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.49996.log.gz 2024-09-11T09:16:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44549.log 2024-09-11T09:16:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49396.log 2024-09-11T09:16:43.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52889.log: /var/log/ceph/cluster2-client.mirror.44549.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52889.log.gz 2024-09-11T09:16:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39490.log 2024-09-11T09:16:43.286 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49396.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.44549.log.gz 2024-09-11T09:16:43.287 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49396.log.gz 2024-09-11T09:16:43.287 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60060.log 2024-09-11T09:16:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40053.log 2024-09-11T09:16:43.288 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39490.log: /var/log/ceph/cluster1-client.mirror.60060.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60060.log.gz 2024-09-11T09:16:43.289 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39490.log.gz 2024-09-11T09:16:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43074.log 2024-09-11T09:16:43.289 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49758.log 2024-09-11T09:16:43.290 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40053.log: /var/log/ceph/cluster2-client.mirror.43074.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43074.log.gz 2024-09-11T09:16:43.290 INFO:teuthology.orchestra.run.smithi077.stderr: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40053.log.gz 2024-09-11T09:16:43.290 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63854.log 2024-09-11T09:16:43.291 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53283.log 2024-09-11T09:16:43.291 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49758.log: /var/log/ceph/cluster2-client.mirror.63854.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.49758.log.gz 2024-09-11T09:16:43.292 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51601.log 2024-09-11T09:16:43.292 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53283.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53283.log.gz 2024-09-11T09:16:43.293 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46068.log 2024-09-11T09:16:43.294 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51601.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63854.log.gz 2024-09-11T09:16:43.294 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51601.log.gz 2024-09-11T09:16:43.294 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51197.log 2024-09-11T09:16:43.294 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66670.log 2024-09-11T09:16:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46068.log: /var/log/ceph/cluster1-client.mirror.51197.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46068.log.gz 2024-09-11T09:16:43.296 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.51197.log.gz 2024-09-11T09:16:43.296 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47401.log 2024-09-11T09:16:43.297 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66670.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66670.log.gz 2024-09-11T09:16:43.297 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39532.log 2024-09-11T09:16:43.297 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36874.log 2024-09-11T09:16:43.298 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47401.log: /var/log/ceph/cluster2-client.mirror.39532.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.47401.log.gz 2024-09-11T09:16:43.298 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41461.log 2024-09-11T09:16:43.299 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36874.log: 90.1% -- replaced with /var/log/ceph/cluster2-client.mirror.36874.log.gz 2024-09-11T09:16:43.300 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57682.log 2024-09-11T09:16:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41461.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.41461.log.gz 2024-09-11T09:16:43.301 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54472.log 2024-09-11T09:16:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57682.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57682.log.gz 2024-09-11T09:16:43.302 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42995.log 2024-09-11T09:16:43.303 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54472.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.54472.log.gz 2024-09-11T09:16:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38483.log 2024-09-11T09:16:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42995.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42995.log.gz 2024-09-11T09:16:43.304 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44366.log 2024-09-11T09:16:43.305 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38483.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.38483.log.gz 2024-09-11T09:16:43.306 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47442.log 2024-09-11T09:16:43.307 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44366.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.44366.log.gz 2024-09-11T09:16:43.308 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67086.log 2024-09-11T09:16:43.310 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47442.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.47442.log.gz 2024-09-11T09:16:43.310 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.25457.log 2024-09-11T09:16:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.67086.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67086.log.gz 2024-09-11T09:16:43.311 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57351.log 2024-09-11T09:16:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.25457.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.25457.log.gz 2024-09-11T09:16:43.312 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65820.log 2024-09-11T09:16:43.314 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57351.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57351.log.gz 2024-09-11T09:16:43.315 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39473.log 2024-09-11T09:16:43.328 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65820.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35169.log 2024-09-11T09:16:43.329 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39473.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39473.log.gz 2024-09-11T09:16:43.332 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39317.log 2024-09-11T09:16:43.333 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35169.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.35169.log.gz 2024-09-11T09:16:43.340 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40440.log.gz 2024-09-11T09:16:43.344 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64242.log 2024-09-11T09:16:43.346 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39317.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39317.log.gz 2024-09-11T09:16:43.346 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46943.log 2024-09-11T09:16:43.347 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64242.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64242.log.gz 2024-09-11T09:16:43.347 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40339.log 2024-09-11T09:16:43.348 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46943.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.46943.log.gz 2024-09-11T09:16:43.349 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.43905.log 2024-09-11T09:16:43.350 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40339.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.40339.log.gz 2024-09-11T09:16:43.350 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51132.log 2024-09-11T09:16:43.364 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.43905.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.35682.log 2024-09-11T09:16:43.365 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51132.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51132.log.gz 2024-09-11T09:16:43.372 INFO:teuthology.orchestra.run.smithi077.stderr: 93.2% -- replaced with /var/log/ceph/cluster1-client.mirror.0.43905.log.gz 2024-09-11T09:16:43.373 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48706.log 2024-09-11T09:16:43.373 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.35682.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41503.log.gz 2024-09-11T09:16:43.374 INFO:teuthology.orchestra.run.smithi077.stderr: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.35682.log.gz 2024-09-11T09:16:43.374 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46919.log 2024-09-11T09:16:43.374 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42111.log 2024-09-11T09:16:43.375 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48706.log: /var/log/ceph/cluster1-client.mirror.46919.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46919.log.gz 2024-09-11T09:16:43.376 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33633.log 2024-09-11T09:16:43.377 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48706.log.gz 2024-09-11T09:16:43.392 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42111.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59228.log 2024-09-11T09:16:43.394 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33633.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.33633.log.gz 2024-09-11T09:16:43.394 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43804.log 2024-09-11T09:16:43.395 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59228.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.59228.log.gz 2024-09-11T09:16:43.395 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49710.log 2024-09-11T09:16:43.409 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43804.log: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.34674.log.gz 2024-09-11T09:16:43.410 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65608.log 2024-09-11T09:16:43.411 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49710.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.49710.log.gz 2024-09-11T09:16:43.411 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55328.log 2024-09-11T09:16:43.412 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65608.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.65608.log.gz 2024-09-11T09:16:43.413 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58075.log 2024-09-11T09:16:43.413 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55328.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.55328.log.gz 2024-09-11T09:16:43.414 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58649.log 2024-09-11T09:16:43.414 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58075.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.58075.log.gz 2024-09-11T09:16:43.415 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42089.log 2024-09-11T09:16:43.416 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58649.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58649.log.gz 2024-09-11T09:16:43.416 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37016.log 2024-09-11T09:16:43.417 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42089.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42089.log.gz 2024-09-11T09:16:43.417 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49123.log 2024-09-11T09:16:43.418 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37016.log: 81.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37016.log.gz 2024-09-11T09:16:43.418 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46389.log 2024-09-11T09:16:43.419 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49123.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49123.log.gz 2024-09-11T09:16:43.420 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47247.log 2024-09-11T09:16:43.421 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46389.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46389.log.gz 2024-09-11T09:16:43.421 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.30823.log 2024-09-11T09:16:43.422 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47247.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.47247.log.gz 2024-09-11T09:16:43.423 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57269.log 2024-09-11T09:16:43.436 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.30823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45027.log 2024-09-11T09:16:43.438 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57269.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57269.log.gz 2024-09-11T09:16:43.438 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.37016.log 2024-09-11T09:16:43.439 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45027.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.45027.log.gz 2024-09-11T09:16:43.455 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39532.log.gz 2024-09-11T09:16:43.456 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34198.log 2024-09-11T09:16:43.459 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.1.37016.log: 92.0% -- replaced with /var/log/ceph/cluster2-client.mirror.1.37016.log.gz 2024-09-11T09:16:43.459 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31306.log 2024-09-11T09:16:43.465 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34198.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34198.log.gz 2024-09-11T09:16:43.465 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54185.log 2024-09-11T09:16:43.466 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.31306.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.31306.log.gz 2024-09-11T09:16:43.466 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51156.log 2024-09-11T09:16:43.467 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54185.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54185.log.gz 2024-09-11T09:16:43.468 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39631.log 2024-09-11T09:16:43.469 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51156.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.51156.log.gz 2024-09-11T09:16:43.470 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33975.log 2024-09-11T09:16:43.470 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39631.log: 79.7% -- replaced with /var/log/ceph/cluster2-client.mirror.39631.log.gz 2024-09-11T09:16:43.470 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54414.log 2024-09-11T09:16:43.472 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33975.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.33975.log.gz 2024-09-11T09:16:43.472 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53192.log 2024-09-11T09:16:43.473 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54414.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54414.log.gz 2024-09-11T09:16:43.474 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34017.log 2024-09-11T09:16:43.476 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53192.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.53192.log.gz 2024-09-11T09:16:43.476 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38646.log 2024-09-11T09:16:43.478 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34017.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.34017.log.gz 2024-09-11T09:16:43.478 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33134.log 2024-09-11T09:16:43.479 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.38646.log: 86.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38646.log.gz 2024-09-11T09:16:43.479 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60757.log 2024-09-11T09:16:43.480 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33134.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.33134.log.gz 2024-09-11T09:16:43.480 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57579.log 2024-09-11T09:16:43.481 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60757.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60757.log.gz 2024-09-11T09:16:43.481 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58897.log 2024-09-11T09:16:43.482 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57579.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57579.log.gz 2024-09-11T09:16:43.483 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40227.log 2024-09-11T09:16:43.484 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58897.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58897.log.gz 2024-09-11T09:16:43.484 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61207.log 2024-09-11T09:16:43.485 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40227.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40227.log.gz 2024-09-11T09:16:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.2.42827.log 2024-09-11T09:16:43.486 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61207.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61207.log.gz 2024-09-11T09:16:43.487 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57599.log 2024-09-11T09:16:43.495 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.2.42827.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37687.log 2024-09-11T09:16:43.504 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57599.log: 90.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41272.log 2024-09-11T09:16:43.504 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57599.log.gz 2024-09-11T09:16:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37687.log: 93.5% 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40789.log.gz 2024-09-11T09:16:43.512 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.37687.log.gz 2024-09-11T09:16:43.512 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56690.log 2024-09-11T09:16:43.513 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41272.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41272.log.gz 2024-09-11T09:16:43.513 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51489.log 2024-09-11T09:16:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56690.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56690.log.gz 2024-09-11T09:16:43.514 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59021.log 2024-09-11T09:16:43.517 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51489.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.51489.log.gz 2024-09-11T09:16:43.517 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33701.log 2024-09-11T09:16:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59021.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59021.log.gz 2024-09-11T09:16:43.518 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52706.log 2024-09-11T09:16:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33701.log: 88.2% -- replaced with /var/log/ceph/cluster1-client.mirror.33701.log.gz 2024-09-11T09:16:43.519 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37988.log 2024-09-11T09:16:43.520 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52706.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.52706.log.gz 2024-09-11T09:16:43.521 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35531.log 2024-09-11T09:16:43.526 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37988.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37988.log.gz 2024-09-11T09:16:43.527 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53982.log 2024-09-11T09:16:43.532 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35531.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.35531.log.gz 2024-09-11T09:16:43.533 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53528.log 2024-09-11T09:16:43.534 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53982.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.53982.log.gz 2024-09-11T09:16:43.534 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58434.log 2024-09-11T09:16:43.535 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53528.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.53528.log.gz 2024-09-11T09:16:43.536 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65798.log 2024-09-11T09:16:43.538 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58434.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58434.log.gz 2024-09-11T09:16:43.538 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57331.log 2024-09-11T09:16:43.539 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65798.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65798.log.gz 2024-09-11T09:16:43.539 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.48321.log 2024-09-11T09:16:43.540 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57331.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57331.log.gz 2024-09-11T09:16:43.540 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60037.log 2024-09-11T09:16:43.542 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.48321.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64626.log.gz 2024-09-11T09:16:43.556 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55927.log 2024-09-11T09:16:43.557 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60037.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60037.log.gz 2024-09-11T09:16:43.558 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57517.log 2024-09-11T09:16:43.558 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55927.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.55927.log.gz 2024-09-11T09:16:43.558 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65058.log 2024-09-11T09:16:43.560 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57517.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57517.log.gz 2024-09-11T09:16:43.560 INFO:teuthology.orchestra.run.smithi077.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.2.42827.log.gz 2024-09-11T09:16:43.560 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63875.log 2024-09-11T09:16:43.561 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65058.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51878.log 2024-09-11T09:16:43.562 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63875.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65058.log.gz 2024-09-11T09:16:43.562 INFO:teuthology.orchestra.run.smithi077.stderr: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.63875.log.gz 2024-09-11T09:16:43.562 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57868.log 2024-09-11T09:16:43.562 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51878.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51878.log.gz 2024-09-11T09:16:43.562 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40488.log 2024-09-11T09:16:43.563 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57868.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55975.log 2024-09-11T09:16:43.563 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40488.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57868.log.gz 2024-09-11T09:16:43.563 INFO:teuthology.orchestra.run.smithi077.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40488.log.gz 2024-09-11T09:16:43.563 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57558.log 2024-09-11T09:16:43.563 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55975.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58525.log 2024-09-11T09:16:43.564 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57558.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.55975.log.gz 2024-09-11T09:16:43.565 INFO:teuthology.orchestra.run.smithi077.stderr: 88.2%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47353.log 2024-09-11T09:16:43.565 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.57558.log.gz 2024-09-11T09:16:43.565 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58525.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58525.log.gz 2024-09-11T09:16:43.566 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42649.log 2024-09-11T09:16:43.566 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47353.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43387.log 2024-09-11T09:16:43.566 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42649.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47353.log.gz 2024-09-11T09:16:43.566 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57083.log 2024-09-11T09:16:43.567 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43387.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43804.log.gz 2024-09-11T09:16:43.573 INFO:teuthology.orchestra.run.smithi077.stderr: 94.4% -- replaced with /var/log/ceph/cluster2-client.mirror.0.48321.log.gz 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42649.log.gz 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39510.log 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46413.log 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57083.log: /var/log/ceph/cluster2-client.mirror.39510.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40361.log 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57083.log.gz 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.39510.log.gz 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46413.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39791.log 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46413.log.gz 2024-09-11T09:16:43.578 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40361.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45095.log 2024-09-11T09:16:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48854.log 2024-09-11T09:16:43.579 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39791.log: /var/log/ceph/cluster1-client.mirror.45095.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45095.log.gz 2024-09-11T09:16:43.591 INFO:teuthology.orchestra.run.smithi077.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.39791.log.gz 2024-09-11T09:16:43.591 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61162.log 2024-09-11T09:16:43.592 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32501.log 2024-09-11T09:16:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61162.log: /var/log/ceph/cluster1-client.mirror.48854.log: 86.8% 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.48854.log.gz 2024-09-11T09:16:43.593 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.61162.log.gz 2024-09-11T09:16:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64371.log 2024-09-11T09:16:43.593 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55162.log 2024-09-11T09:16:43.594 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32501.log: /var/log/ceph/cluster2-client.mirror.64371.log: 76.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64371.log.gz 2024-09-11T09:16:43.595 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32501.log.gz 2024-09-11T09:16:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38283.log 2024-09-11T09:16:43.595 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54631.log 2024-09-11T09:16:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.55162.log: /var/log/ceph/cluster2-client.mirror.38283.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38283.log.gz 2024-09-11T09:16:43.596 INFO:teuthology.orchestra.run.smithi077.stderr: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.55162.log.gz 2024-09-11T09:16:43.596 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41602.log 2024-09-11T09:16:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.42649.log 2024-09-11T09:16:43.597 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54631.log: /var/log/ceph/cluster2-client.mirror.41602.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41602.log.gz 2024-09-11T09:16:43.597 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.54631.log.gz 2024-09-11T09:16:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43170.log 2024-09-11T09:16:43.598 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66982.log 2024-09-11T09:16:43.599 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.42649.log: /var/log/ceph/cluster2-client.mirror.43170.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43170.log.gz 2024-09-11T09:16:43.601 INFO:teuthology.orchestra.run.smithi077.stderr: 94.0% -- replaced with /var/log/ceph/cluster2-client.mirror.30823.log.gz 2024-09-11T09:16:43.601 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42111.log.gz 2024-09-11T09:16:43.604 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54520.log 2024-09-11T09:16:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.66982.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60127.log 2024-09-11T09:16:43.608 INFO:teuthology.orchestra.run.smithi077.stderr: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66982.log.gz 2024-09-11T09:16:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:gzip/var/log/ceph/cluster1-client.mirror.54520.log: -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60555.log 2024-09-11T09:16:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60127.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40381.log 2024-09-11T09:16:43.608 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.54520.log.gz 2024-09-11T09:16:43.608 INFO:teuthology.orchestra.run.smithi077.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60127.log.gzgzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43365.log 2024-09-11T09:16:43.608 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60555.log: 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60555.log.gz 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40381.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.55473.log 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40381.log.gz 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63162.log 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43365.log: gzip -5 --verbose --/var/log/ceph/cluster2-client.mirror.55473.log: /var/log/ceph/cluster2-client.mirror.39611.log 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.43365.log.gz 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr: 87.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33814.log 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.55473.log.gz 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63162.log: /var/log/ceph/cluster2-client.mirror.39611.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.63162.log.gz 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40269.log 2024-09-11T09:16:43.609 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33814.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62158.log 2024-09-11T09:16:43.613 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40269.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.33814.log.gz 2024-09-11T09:16:43.614 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.33382.log 2024-09-11T09:16:43.618 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.62158.log: 93.1% -- replaced with /var/log/ceph/cluster1-client.mirror.0.42649.log.gz 2024-09-11T09:16:43.620 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32435.log 2024-09-11T09:16:43.621 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.33382.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.admin.33382.log.gz 2024-09-11T09:16:43.621 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41125.log 2024-09-11T09:16:43.623 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32435.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32435.log.gz 2024-09-11T09:16:43.623 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40959.log 2024-09-11T09:16:43.636 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41125.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.52417.log 2024-09-11T09:16:43.648 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40959.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32851.log 2024-09-11T09:16:43.650 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.52417.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.52417.log.gz 2024-09-11T09:16:43.656 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65820.log.gz 2024-09-11T09:16:43.664 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51444.log 2024-09-11T09:16:43.680 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32851.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57971.log 2024-09-11T09:16:43.681 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51444.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.51444.log.gz 2024-09-11T09:16:43.696 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49186.log 2024-09-11T09:16:43.699 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57971.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57971.log.gz 2024-09-11T09:16:43.699 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60397.log 2024-09-11T09:16:43.700 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49186.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.49186.log.gz 2024-09-11T09:16:43.716 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61680.log 2024-09-11T09:16:43.718 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60397.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60397.log.gz 2024-09-11T09:16:43.730 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41775.log 2024-09-11T09:16:43.731 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61680.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61680.log.gz 2024-09-11T09:16:43.744 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30313.log 2024-09-11T09:16:43.753 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41775.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53873.log 2024-09-11T09:16:43.754 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30313.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30313.log.gz 2024-09-11T09:16:43.773 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47534.log 2024-09-11T09:16:43.773 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53873.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53873.log.gz 2024-09-11T09:16:43.788 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36603.log 2024-09-11T09:16:43.789 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47534.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.47534.log.gz 2024-09-11T09:16:43.796 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.62081.log 2024-09-11T09:16:43.806 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36603.log: 93.5% 93.9%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63337.log 2024-09-11T09:16:43.807 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.62081.log: 2.6% -- replaced with /var/log/ceph/cluster2-client.mirror.62081.log.gz 2024-09-11T09:16:43.807 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40269.log.gz 2024-09-11T09:16:43.816 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.36603.log.gz 2024-09-11T09:16:43.816 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65888.log 2024-09-11T09:16:43.819 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63337.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.63337.log.gz 2024-09-11T09:16:43.832 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64739.log 2024-09-11T09:16:43.833 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9%/var/log/ceph/cluster1-client.mirror.65888.log: -- replaced with /var/log/ceph/cluster2-client.mirror.40959.log.gz 2024-09-11T09:16:43.833 INFO:teuthology.orchestra.run.smithi077.stderr: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.65888.log.gz 2024-09-11T09:16:43.836 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65930.log 2024-09-11T09:16:43.837 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64739.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64739.log.gz 2024-09-11T09:16:43.856 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35084.log 2024-09-11T09:16:43.857 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65930.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.65930.log.gz 2024-09-11T09:16:43.861 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.62158.log.gz 2024-09-11T09:16:43.867 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43387.log.gz 2024-09-11T09:16:43.868 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40398.log 2024-09-11T09:16:43.868 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35084.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59290.log 2024-09-11T09:16:43.870 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40398.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40398.log.gz 2024-09-11T09:16:43.870 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34241.log 2024-09-11T09:16:43.871 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59290.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59290.log.gz 2024-09-11T09:16:43.871 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66037.log 2024-09-11T09:16:43.873 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34241.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.35084.log.gz 2024-09-11T09:16:43.876 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.34241.log.gz 2024-09-11T09:16:43.877 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57765.log 2024-09-11T09:16:43.878 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39448.log 2024-09-11T09:16:43.878 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66037.log: /var/log/ceph/cluster2-client.mirror.57765.log: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66037.log.gz 2024-09-11T09:16:43.878 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.57765.log.gz 2024-09-11T09:16:43.878 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51830.log 2024-09-11T09:16:43.888 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39448.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61927.log 2024-09-11T09:16:43.889 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51830.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51830.log.gz 2024-09-11T09:16:43.889 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59600.log 2024-09-11T09:16:43.890 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61927.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61927.log.gz 2024-09-11T09:16:43.891 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59104.log 2024-09-11T09:16:43.892 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59600.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59600.log.gz 2024-09-11T09:16:43.892 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40917.log 2024-09-11T09:16:43.893 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59104.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39611.log.gz 2024-09-11T09:16:43.893 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59104.log.gz 2024-09-11T09:16:43.894 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41331.log 2024-09-11T09:16:43.894 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40917.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35682.log 2024-09-11T09:16:43.894 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41331.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40917.log.gz 2024-09-11T09:16:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33334.log 2024-09-11T09:16:43.895 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35682.log: 81.7% -- replaced with /var/log/ceph/cluster2-client.mirror.35682.log.gz 2024-09-11T09:16:43.896 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36645.log 2024-09-11T09:16:43.904 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33334.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.34812.log 2024-09-11T09:16:43.905 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36645.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36645.log.gz 2024-09-11T09:16:43.920 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39256.log 2024-09-11T09:16:43.939 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.2.34812.log: 92.9%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31204.log 2024-09-11T09:16:43.944 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39256.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56224.log 2024-09-11T09:16:43.952 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50144.log 2024-09-11T09:16:43.953 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56224.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56224.log.gz 2024-09-11T09:16:43.961 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53172.log 2024-09-11T09:16:43.961 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31204.log: 2.3% -- replaced with /var/log/ceph/cluster1-client.mirror.31204.log.gz/var/log/ceph/cluster1-client.mirror.50144.log: 2024-09-11T09:16:43.961 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50144.log.gz 2024-09-11T09:16:43.968 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.2.34812.log.gz 2024-09-11T09:16:43.976 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65332.log 2024-09-11T09:16:43.978 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53172.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.53172.log.gz 2024-09-11T09:16:43.988 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38243.log 2024-09-11T09:16:43.989 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65332.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65332.log.gz 2024-09-11T09:16:44.000 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41125.log.gz 2024-09-11T09:16:44.000 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35553.log 2024-09-11T09:16:44.001 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38243.log: 88.6% -- replaced with /var/log/ceph/cluster2-client.mirror.38243.log.gz 2024-09-11T09:16:44.002 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65143.log 2024-09-11T09:16:44.003 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35553.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.35553.log.gz 2024-09-11T09:16:44.003 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41753.log 2024-09-11T09:16:44.006 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65143.log: 91.6% -- replaced with /var/log/ceph/cluster2-client.mirror.65143.log.gz 2024-09-11T09:16:44.006 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47514.log 2024-09-11T09:16:44.007 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41753.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41753.log.gz 2024-09-11T09:16:44.007 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56793.log 2024-09-11T09:16:44.009 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47514.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.47514.log.gz 2024-09-11T09:16:44.009 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64484.log 2024-09-11T09:16:44.012 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56793.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56793.log.gz 2024-09-11T09:16:44.012 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32479.log 2024-09-11T09:16:44.013 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64484.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.64484.log.gz 2024-09-11T09:16:44.013 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42069.log 2024-09-11T09:16:44.014 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32479.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32479.log.gz 2024-09-11T09:16:44.014 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57847.log 2024-09-11T09:16:44.016 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42069.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42069.log.gz 2024-09-11T09:16:44.016 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.66904.log 2024-09-11T09:16:44.019 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57847.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57847.log.gz 2024-09-11T09:16:44.019 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58959.log 2024-09-11T09:16:44.020 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.66904.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.66904.log.gz 2024-09-11T09:16:44.020 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42933.log 2024-09-11T09:16:44.021 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58959.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58959.log.gz 2024-09-11T09:16:44.021 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65522.log 2024-09-11T09:16:44.023 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42933.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42933.log.gz 2024-09-11T09:16:44.023 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44798.log 2024-09-11T09:16:44.024 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65522.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.65522.log.gz 2024-09-11T09:16:44.024 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38604.log 2024-09-11T09:16:44.025 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44798.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44798.log.gz 2024-09-11T09:16:44.025 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57475.log 2024-09-11T09:16:44.026 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38604.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38604.log.gz 2024-09-11T09:16:44.027 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38694.log 2024-09-11T09:16:44.029 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57475.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57475.log.gz 2024-09-11T09:16:44.030 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56855.log 2024-09-11T09:16:44.030 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38694.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38694.log.gz 2024-09-11T09:16:44.031 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45119.log 2024-09-11T09:16:44.033 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56855.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.56855.log.gz 2024-09-11T09:16:44.033 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34085.log 2024-09-11T09:16:44.034 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45119.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.45119.log.gz 2024-09-11T09:16:44.035 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52059.log 2024-09-11T09:16:44.036 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34085.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.34085.log.gz 2024-09-11T09:16:44.036 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58670.log 2024-09-11T09:16:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52059.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52059.log.gz 2024-09-11T09:16:44.037 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40319.log 2024-09-11T09:16:44.038 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58670.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58670.log.gz 2024-09-11T09:16:44.038 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.66387.log 2024-09-11T09:16:44.040 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40319.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.40319.log.gz 2024-09-11T09:16:44.040 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65223.log 2024-09-11T09:16:44.041 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.66387.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.66387.log.gz 2024-09-11T09:16:44.041 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48806.log 2024-09-11T09:16:44.060 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65223.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46548.log 2024-09-11T09:16:44.061 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48806.log: 87.4% -- replaced with /var/log/ceph/cluster2-client.mirror.48806.log.gz 2024-09-11T09:16:44.062 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40361.log.gz 2024-09-11T09:16:44.062 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50016.log 2024-09-11T09:16:44.064 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46548.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.46548.log.gz 2024-09-11T09:16:44.076 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.30949.log 2024-09-11T09:16:44.078 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50016.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50016.log.gz 2024-09-11T09:16:44.084 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40631.log 2024-09-11T09:16:44.091 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.30949.log: 95.5% 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41331.log.gz 2024-09-11T09:16:44.096 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.30949.log.gz 2024-09-11T09:16:44.096 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31238.log 2024-09-11T09:16:44.104 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40631.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.52728.log 2024-09-11T09:16:44.105 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31238.log: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31238.log.gz 2024-09-11T09:16:44.112 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61702.log 2024-09-11T09:16:44.117 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.52728.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39256.log.gz 2024-09-11T09:16:44.128 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64993.log 2024-09-11T09:16:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61702.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.61702.log.gz 2024-09-11T09:16:44.129 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59558.log 2024-09-11T09:16:44.130 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64993.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.64993.log.gz 2024-09-11T09:16:44.144 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59857.log 2024-09-11T09:16:44.147 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59558.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.59558.log.gz 2024-09-11T09:16:44.148 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47088.log 2024-09-11T09:16:44.148 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59857.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59857.log.gz 2024-09-11T09:16:44.159 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.50255.log 2024-09-11T09:16:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47088.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.47088.log.gz 2024-09-11T09:16:44.161 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster1-client.mirror.0.52728.log.gz 2024-09-11T09:16:44.161 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58262.log 2024-09-11T09:16:44.162 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.50255.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.50255.log.gz 2024-09-11T09:16:44.163 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59145.log 2024-09-11T09:16:44.164 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58262.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58262.log.gz 2024-09-11T09:16:44.165 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64262.log 2024-09-11T09:16:44.166 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59145.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59145.log.gz 2024-09-11T09:16:44.166 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37169.log 2024-09-11T09:16:44.167 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64262.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.64262.log.gz 2024-09-11T09:16:44.167 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59925.log 2024-09-11T09:16:44.173 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37169.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37169.log.gz 2024-09-11T09:16:44.173 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42259.log 2024-09-11T09:16:44.174 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59925.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.59925.log.gz 2024-09-11T09:16:44.174 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44431.log 2024-09-11T09:16:44.175 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42259.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.42259.log.gz 2024-09-11T09:16:44.176 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43111.log 2024-09-11T09:16:44.179 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44431.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44431.log.gz 2024-09-11T09:16:44.180 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49422.log 2024-09-11T09:16:44.180 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43111.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.43111.log.gz 2024-09-11T09:16:44.181 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34474.log 2024-09-11T09:16:44.196 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49422.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47422.log 2024-09-11T09:16:44.197 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34474.log: 88.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34474.log.gz 2024-09-11T09:16:44.210 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42439.log 2024-09-11T09:16:44.211 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47422.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.47422.log.gz 2024-09-11T09:16:44.211 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.49422.log.gz 2024-09-11T09:16:44.212 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56023.log 2024-09-11T09:16:44.232 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42439.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58393.log 2024-09-11T09:16:44.233 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56023.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.56023.log.gz 2024-09-11T09:16:44.238 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39448.log.gz 2024-09-11T09:16:44.239 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34498.log 2024-09-11T09:16:44.240 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58393.log: 88.1% 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41775.log.gz 2024-09-11T09:16:44.240 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.58393.log.gz 2024-09-11T09:16:44.240 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65432.log 2024-09-11T09:16:44.241 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34498.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.48145.log 2024-09-11T09:16:44.241 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65432.log: 88.3% -- replaced with /var/log/ceph/cluster1-client.mirror.34498.log.gz 2024-09-11T09:16:44.242 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51468.log 2024-09-11T09:16:44.242 INFO:teuthology.orchestra.run.smithi077.stderr: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.65432.log.gz 2024-09-11T09:16:44.243 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.48145.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.48145.log.gz 2024-09-11T09:16:44.244 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46895.log 2024-09-11T09:16:44.244 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51468.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54163.log 2024-09-11T09:16:44.245 INFO:teuthology.orchestra.run.smithi077.stderr: 88.0%/var/log/ceph/cluster1-client.mirror.46895.log: -- replaced with /var/log/ceph/cluster1-client.mirror.51468.log.gz 2024-09-11T09:16:44.245 INFO:teuthology.orchestra.run.smithi077.stderr: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.46895.log.gz 2024-09-11T09:16:44.245 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56474.log 2024-09-11T09:16:44.247 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.54163.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.54163.log.gz 2024-09-11T09:16:44.247 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56835.log 2024-09-11T09:16:44.249 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56474.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41661.log 2024-09-11T09:16:44.249 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56835.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56474.log.gz 2024-09-11T09:16:44.249 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.56835.log.gzgzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44665.log 2024-09-11T09:16:44.249 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:44.264 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41661.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46506.log 2024-09-11T09:16:44.266 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44665.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.44665.log.gz 2024-09-11T09:16:44.266 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38731.log 2024-09-11T09:16:44.268 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46506.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.46506.log.gz 2024-09-11T09:16:44.268 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48269.log 2024-09-11T09:16:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38731.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.38731.log.gz 2024-09-11T09:16:44.269 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46827.log 2024-09-11T09:16:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48269.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48269.log.gz 2024-09-11T09:16:44.270 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40547.log 2024-09-11T09:16:44.272 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46827.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.46827.log.gz 2024-09-11T09:16:44.272 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.35861.log 2024-09-11T09:16:44.275 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40547.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.65223.log.gz 2024-09-11T09:16:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.32457.log 2024-09-11T09:16:44.276 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.35861.log: 89.0% -- replaced with /var/log/ceph/cluster1-client.mirror.35861.log.gz 2024-09-11T09:16:44.277 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48321.log 2024-09-11T09:16:44.278 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.32457.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.admin.32457.log.gz 2024-09-11T09:16:44.278 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64839.log 2024-09-11T09:16:44.280 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48321.log: 95.1% -- replaced with /var/log/ceph/cluster1-client.mirror.48321.log.gz 2024-09-11T09:16:44.280 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44939.log 2024-09-11T09:16:44.281 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64839.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64839.log.gz 2024-09-11T09:16:44.282 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56426.log 2024-09-11T09:16:44.284 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44939.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.44939.log.gz 2024-09-11T09:16:44.284 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49951.log 2024-09-11T09:16:44.285 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56426.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.56426.log.gz 2024-09-11T09:16:44.285 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37472.log 2024-09-11T09:16:44.286 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49951.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.49951.log.gz 2024-09-11T09:16:44.286 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.0.45252.log 2024-09-11T09:16:44.287 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37472.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32851.log.gz 2024-09-11T09:16:44.290 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40631.log.gz 2024-09-11T09:16:44.292 INFO:teuthology.orchestra.run.smithi077.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37472.log.gz 2024-09-11T09:16:44.292 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61567.log 2024-09-11T09:16:44.293 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.0.45252.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52934.log 2024-09-11T09:16:44.293 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61567.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56152.log 2024-09-11T09:16:44.294 INFO:teuthology.orchestra.run.smithi077.stderr: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61567.log.gz 2024-09-11T09:16:44.295 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52934.log: 92.4% -- replaced with /var/log/ceph/cluster2-client.mirror.52934.log.gz 2024-09-11T09:16:44.296 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.51331.log 2024-09-11T09:16:44.296 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56152.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51509.log 2024-09-11T09:16:44.296 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%/var/log/ceph/cluster2-client.mirror.3.51331.log: -- replaced with /var/log/ceph/cluster2-client.mirror.56152.log.gz 2024-09-11T09:16:44.312 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36894.log 2024-09-11T09:16:44.314 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51509.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.51509.log.gz 2024-09-11T09:16:44.315 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45999.log 2024-09-11T09:16:44.316 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36894.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.36894.log.gz 2024-09-11T09:16:44.316 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59902.log 2024-09-11T09:16:44.317 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45999.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.45999.log.gz 2024-09-11T09:16:44.317 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64505.log 2024-09-11T09:16:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59902.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.59902.log.gz 2024-09-11T09:16:44.318 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31272.log 2024-09-11T09:16:44.319 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64505.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.64505.log.gz 2024-09-11T09:16:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.31255.log 2024-09-11T09:16:44.320 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.33334.log.gz 2024-09-11T09:16:44.320 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31272.log: 54.2% -- replaced with /var/log/ceph/cluster1-client.mirror.31272.log.gz 2024-09-11T09:16:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47984.log 2024-09-11T09:16:44.321 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.31255.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46485.log 2024-09-11T09:16:44.321 INFO:teuthology.orchestra.run.smithi077.stderr: 57.4% -- replaced with /var/log/ceph/cluster1-client.mirror.31255.log.gz 2024-09-11T09:16:44.322 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47984.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.47984.log.gz 2024-09-11T09:16:44.323 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.28057.log 2024-09-11T09:16:44.323 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38074.log 2024-09-11T09:16:44.324 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46485.log: /var/log/ceph/cluster2-client.admin.28057.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.28057.log.gz 2024-09-11T09:16:44.324 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46485.log.gz 2024-09-11T09:16:44.324 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35813.log 2024-09-11T09:16:44.324 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52481.log 2024-09-11T09:16:44.325 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38074.log: /var/log/ceph/cluster2-client.mirror.35813.log: 89.2% -- replaced with /var/log/ceph/cluster2-client.mirror.35813.log.gz 2024-09-11T09:16:44.327 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.3.51331.log.gz 2024-09-11T09:16:44.329 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.38074.log.gz 2024-09-11T09:16:44.330 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.65671.log 2024-09-11T09:16:44.330 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33408.log 2024-09-11T09:16:44.330 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52481.log: /var/log/ceph/cluster1-client.mirror.2.65671.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.33773.log 2024-09-11T09:16:44.331 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33408.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.52481.log.gz 2024-09-11T09:16:44.331 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.41355.log 2024-09-11T09:16:44.332 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.33773.log: 91.8% -- replaced with /var/log/ceph/cluster2-client.mirror.33408.log.gz 2024-09-11T09:16:44.332 INFO:teuthology.orchestra.run.smithi077.stderr: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.33773.log.gz 2024-09-11T09:16:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61005.log 2024-09-11T09:16:44.334 INFO:teuthology.orchestra.run.smithi077.stderr: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.2.65671.log.gz 2024-09-11T09:16:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39128.log 2024-09-11T09:16:44.334 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.41355.log: /var/log/ceph/cluster1-client.mirror.61005.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.65761.log 2024-09-11T09:16:44.334 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7%/var/log/ceph/cluster2-client.mirror.39128.log: -- replaced with /var/log/ceph/cluster1-client.mirror.61005.log.gz 2024-09-11T09:16:44.347 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.0.45252.log.gz 2024-09-11T09:16:44.348 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66054.log 2024-09-11T09:16:44.349 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.65761.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44569.log 2024-09-11T09:16:44.349 INFO:teuthology.orchestra.run.smithi077.stderr: 80.0% -- replaced with /var/log/ceph/cluster2-client.mirror.65761.log.gz 2024-09-11T09:16:44.349 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66054.log: 89.6% -- replaced with /var/log/ceph/cluster2-client.mirror.66054.log.gz 2024-09-11T09:16:44.350 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.2.30834.log 2024-09-11T09:16:44.350 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37406.log 2024-09-11T09:16:44.351 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44569.log: /var/log/ceph/cluster1-client.mirror.2.30834.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44569.log.gz 2024-09-11T09:16:44.351 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42002.log 2024-09-11T09:16:44.355 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37406.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.2.30834.log.gz 2024-09-11T09:16:44.357 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37406.log.gz 2024-09-11T09:16:44.357 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66695.log 2024-09-11T09:16:44.358 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63813.log 2024-09-11T09:16:44.358 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42002.log: /var/log/ceph/cluster2-client.mirror.66695.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.mirror.66695.log.gz 2024-09-11T09:16:44.358 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.42002.log.gz 2024-09-11T09:16:44.359 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61792.log 2024-09-11T09:16:44.359 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57186.log 2024-09-11T09:16:44.360 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63813.log: /var/log/ceph/cluster1-client.mirror.61792.log: 84.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61792.log.gz 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63813.log.gz 2024-09-11T09:16:44.360 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:16:44.360 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.54917.log 2024-09-11T09:16:44.360 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63296.log 2024-09-11T09:16:44.361 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57186.log: /var/log/ceph/cluster1-client.mirror.54917.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57186.log.gz 2024-09-11T09:16:44.361 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.36061.log 2024-09-11T09:16:44.362 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63296.log: 92.2% -- replaced with /var/log/ceph/cluster1-client.mirror.54917.log.gz 2024-09-11T09:16:44.362 INFO:teuthology.orchestra.run.smithi077.stderr: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.63296.log.gz 2024-09-11T09:16:44.363 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44846.log 2024-09-11T09:16:44.363 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30235.log 2024-09-11T09:16:44.364 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.36061.log: /var/log/ceph/cluster1-client.mirror.44846.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.44846.log.gz 2024-09-11T09:16:44.376 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.1.30922.log 2024-09-11T09:16:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30235.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30235.log.gz 2024-09-11T09:16:44.377 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49252.log 2024-09-11T09:16:44.384 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.1.30922.log: 94.1% -- replaced with /var/log/ceph/cluster2-client.mirror.0.41355.log.gz 2024-09-11T09:16:44.384 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.1.30922.log.gz 2024-09-11T09:16:44.384 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59476.log 2024-09-11T09:16:44.384 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60667.log 2024-09-11T09:16:44.385 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49252.log: /var/log/ceph/cluster2-client.mirror.59476.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49252.log.gz 2024-09-11T09:16:44.386 INFO:teuthology.orchestra.run.smithi077.stderr: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.59476.log.gz 2024-09-11T09:16:44.386 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40747.log 2024-09-11T09:16:44.386 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60667.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60667.log.gz 2024-09-11T09:16:44.386 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39359.log 2024-09-11T09:16:44.387 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56450.log 2024-09-11T09:16:44.388 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40747.log: /var/log/ceph/cluster2-client.mirror.39359.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.40747.log.gz 2024-09-11T09:16:44.388 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41540.log 2024-09-11T09:16:44.389 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56450.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.56450.log.gz 2024-09-11T09:16:44.393 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54689.log 2024-09-11T09:16:44.394 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41540.log: 89.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41540.log.gz 2024-09-11T09:16:44.394 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57124.log 2024-09-11T09:16:44.394 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54689.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.54689.log.gz 2024-09-11T09:16:44.394 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61185.log 2024-09-11T09:16:44.394 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57124.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57124.log.gz 2024-09-11T09:16:44.394 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41024.log 2024-09-11T09:16:44.394 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61185.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.61185.log.gz 2024-09-11T09:16:44.395 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.51036.log 2024-09-11T09:16:44.395 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41024.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.41024.log.gz 2024-09-11T09:16:44.395 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61725.log 2024-09-11T09:16:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.51036.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.51036.log.gz 2024-09-11T09:16:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48685.log 2024-09-11T09:16:44.396 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61725.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61725.log.gz 2024-09-11T09:16:44.397 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46240.log 2024-09-11T09:16:44.398 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48685.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.48685.log.gz 2024-09-11T09:16:44.398 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35239.log 2024-09-11T09:16:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46240.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46240.log.gz 2024-09-11T09:16:44.399 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55256.log 2024-09-11T09:16:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.35239.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.admin.35239.log.gz 2024-09-11T09:16:44.400 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37126.log 2024-09-11T09:16:44.401 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55256.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.55256.log.gz 2024-09-11T09:16:44.401 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67112.log 2024-09-11T09:16:44.407 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37126.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37126.log.gz 2024-09-11T09:16:44.407 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64881.log 2024-09-11T09:16:44.408 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.67112.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67112.log.gz 2024-09-11T09:16:44.408 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61050.log 2024-09-11T09:16:44.424 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64881.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.35606.log 2024-09-11T09:16:44.425 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61050.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61050.log.gz 2024-09-11T09:16:44.430 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.40547.log.gz 2024-09-11T09:16:44.431 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42775.log 2024-09-11T09:16:44.432 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.35606.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.admin.35606.log.gz 2024-09-11T09:16:44.432 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46666.log 2024-09-11T09:16:44.433 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42775.log: 87.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42775.log.gz 2024-09-11T09:16:44.433 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37859.log 2024-09-11T09:16:44.434 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46666.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.46666.log.gz 2024-09-11T09:16:44.435 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42755.log 2024-09-11T09:16:44.440 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37859.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.37859.log.gz 2024-09-11T09:16:44.441 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57062.log 2024-09-11T09:16:44.443 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42755.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42755.log.gz 2024-09-11T09:16:44.443 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47855.log 2024-09-11T09:16:44.443 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57062.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.57062.log.gz 2024-09-11T09:16:44.444 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61747.log 2024-09-11T09:16:44.445 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47855.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.47855.log.gz 2024-09-11T09:16:44.445 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41982.log 2024-09-11T09:16:44.446 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61747.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61747.log.gz 2024-09-11T09:16:44.446 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56269.log 2024-09-11T09:16:44.447 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41982.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41982.log.gz 2024-09-11T09:16:44.448 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64695.log 2024-09-11T09:16:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56269.log: 92.7% -- replaced with /var/log/ceph/cluster2-client.mirror.56269.log.gz 2024-09-11T09:16:44.450 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.49662.log 2024-09-11T09:16:44.451 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64695.log: 88.8% -- replaced with /var/log/ceph/cluster2-client.mirror.64695.log.gz 2024-09-11T09:16:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47920.log 2024-09-11T09:16:44.452 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.49662.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.49662.log.gz 2024-09-11T09:16:44.453 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53355.log 2024-09-11T09:16:44.455 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47920.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.47920.log.gz 2024-09-11T09:16:44.455 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40302.log 2024-09-11T09:16:44.456 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53355.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.53355.log.gz 2024-09-11T09:16:44.456 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30399.log 2024-09-11T09:16:44.457 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40302.log: 79.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40302.log.gz 2024-09-11T09:16:44.457 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.34630.log 2024-09-11T09:16:44.458 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30399.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30399.log.gz 2024-09-11T09:16:44.458 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57248.log 2024-09-11T09:16:44.464 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.34630.log: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.34630.log.gz 2024-09-11T09:16:44.465 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32670.log 2024-09-11T09:16:44.466 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57248.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57248.log.gz 2024-09-11T09:16:44.466 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.30771.log 2024-09-11T09:16:44.467 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.32670.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.32670.log.gz 2024-09-11T09:16:44.467 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.0.log 2024-09-11T09:16:44.468 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.30771.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.30771.log.gz 2024-09-11T09:16:44.468 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.45252.log 2024-09-11T09:16:44.480 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-osd.0.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48028.log 2024-09-11T09:16:44.484 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.45252.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42439.log.gz 2024-09-11T09:16:44.496 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.3.30949.log 2024-09-11T09:16:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48028.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.48028.log.gz 2024-09-11T09:16:44.497 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.48637.log 2024-09-11T09:16:44.528 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.3.30949.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.51697.log 2024-09-11T09:16:44.529 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.48637.log: 87.5% -- replaced with /var/log/ceph/cluster1-client.mirror.48637.log.gz 2024-09-11T09:16:44.534 INFO:teuthology.orchestra.run.smithi077.stderr: 93.7% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.41661.log.gz 2024-09-11T09:16:44.534 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61500.log 2024-09-11T09:16:44.535 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.51697.log: 88.1% -- replaced with /var/log/ceph/cluster1-client.mirror.51697.log.gz 2024-09-11T09:16:44.536 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43212.log 2024-09-11T09:16:44.536 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61500.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.61500.log.gz 2024-09-11T09:16:44.536 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37363.log 2024-09-11T09:16:44.552 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43212.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41560.log 2024-09-11T09:16:44.556 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37363.log: -- replaced with /var/log/ceph/cluster2-client.mirror.45252.log.gz 2024-09-11T09:16:44.558 INFO:teuthology.orchestra.run.smithi077.stderr: 93.4% -- replaced with /var/log/ceph/cluster1-client.mirror.37363.log.gz 2024-09-11T09:16:44.564 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49552.log 2024-09-11T09:16:44.565 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41560.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41560.log.gz 2024-09-11T09:16:44.565 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.64881.log.gz 2024-09-11T09:16:44.566 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38463.log 2024-09-11T09:16:44.568 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49552.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.49552.log.gz 2024-09-11T09:16:44.568 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.0.63006.log 2024-09-11T09:16:44.569 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38463.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.38463.log.gz 2024-09-11T09:16:44.570 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.61657.log 2024-09-11T09:16:44.576 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.0.63006.log: 93.5% -- replaced with /var/log/ceph/cluster2-client.mirror.0.63006.log.gz 2024-09-11T09:16:44.576 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.46285.log 2024-09-11T09:16:44.577 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.61657.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.61657.log.gz 2024-09-11T09:16:44.577 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65564.log 2024-09-11T09:16:44.579 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.46285.log: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.46285.log.gz 2024-09-11T09:16:44.579 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53849.log 2024-09-11T09:16:44.580 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65564.log: 85.1% -- replaced with /var/log/ceph/cluster1-client.mirror.65564.log.gz 2024-09-11T09:16:44.581 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50168.log 2024-09-11T09:16:44.581 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53849.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.53849.log.gz 2024-09-11T09:16:44.582 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64801.log 2024-09-11T09:16:44.583 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50168.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.50168.log.gz 2024-09-11T09:16:44.583 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42597.log 2024-09-11T09:16:44.586 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64801.log: 91.7% -- replaced with /var/log/ceph/cluster2-client.mirror.64801.log.gz 2024-09-11T09:16:44.586 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36443.log 2024-09-11T09:16:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42597.log: 94.3% -- replaced with /var/log/ceph/cluster2-client.mirror.3.30949.log.gz 2024-09-11T09:16:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.36491.log 2024-09-11T09:16:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36443.log: 89.4% -- replaced with /var/log/ceph/cluster1-client.mirror.36443.log.gz 2024-09-11T09:16:44.597 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63792.log 2024-09-11T09:16:44.598 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.36491.log: 89.3% -- replaced with /var/log/ceph/cluster1-client.mirror.36491.log.gz 2024-09-11T09:16:44.598 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.62064.log 2024-09-11T09:16:44.599 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63792.log: 87.0% -- replaced with /var/log/ceph/cluster1-client.mirror.63792.log.gz 2024-09-11T09:16:44.599 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58054.log 2024-09-11T09:16:44.600 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.62064.log: 2.6% -- replaced with /var/log/ceph/cluster1-client.mirror.62064.log.gz 2024-09-11T09:16:44.600 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49276.log 2024-09-11T09:16:44.601 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58054.log: 88.0% -- replaced with /var/log/ceph/cluster2-client.mirror.58054.log.gz 2024-09-11T09:16:44.601 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53466.log 2024-09-11T09:16:44.602 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49276.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49276.log.gz 2024-09-11T09:16:44.603 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56132.log 2024-09-11T09:16:44.604 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53466.log: 89.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53466.log.gz 2024-09-11T09:16:44.604 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38363.log 2024-09-11T09:16:44.606 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56132.log: 90.0% -- replaced with /var/log/ceph/cluster2-client.mirror.56132.log.gz 2024-09-11T09:16:44.606 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58690.log 2024-09-11T09:16:44.607 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38363.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.38363.log.gz 2024-09-11T09:16:44.607 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.64028.log 2024-09-11T09:16:44.610 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58690.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.58690.log.gz 2024-09-11T09:16:44.610 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57207.log 2024-09-11T09:16:44.611 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.64028.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.64028.log.gz 2024-09-11T09:16:44.611 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47783.log 2024-09-11T09:16:44.612 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57207.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57207.log.gz 2024-09-11T09:16:44.612 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.47807.log 2024-09-11T09:16:44.613 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47783.log: 87.4% -- replaced with /var/log/ceph/cluster1-client.mirror.47783.log.gz 2024-09-11T09:16:44.614 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39747.log 2024-09-11T09:16:44.615 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.47807.log: 87.6% -- replaced with /var/log/ceph/cluster1-client.mirror.47807.log.gz 2024-09-11T09:16:44.615 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38137.log 2024-09-11T09:16:44.616 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39747.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.39747.log.gz 2024-09-11T09:16:44.616 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53214.log 2024-09-11T09:16:44.622 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.38137.log: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38137.log.gz 2024-09-11T09:16:44.622 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.64093.log 2024-09-11T09:16:44.624 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53214.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.53214.log.gz 2024-09-11T09:16:44.624 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60600.log 2024-09-11T09:16:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.64093.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.64093.log.gz 2024-09-11T09:16:44.625 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59207.log 2024-09-11T09:16:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60600.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.60600.log.gz 2024-09-11T09:16:44.626 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.32256.log 2024-09-11T09:16:44.627 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59207.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.59207.log.gz 2024-09-11T09:16:44.627 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58587.log 2024-09-11T09:16:44.630 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.32256.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39359.log.gz 2024-09-11T09:16:44.640 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.39769.log 2024-09-11T09:16:44.641 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58587.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.58587.log.gz 2024-09-11T09:16:44.641 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.66431.log 2024-09-11T09:16:44.648 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.39769.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60262.log 2024-09-11T09:16:44.661 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.66431.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.66431.log.gz 2024-09-11T09:16:44.664 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.53151.log 2024-09-11T09:16:44.665 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60262.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60262.log.gz 2024-09-11T09:16:44.665 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.47739.log 2024-09-11T09:16:44.666 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.53151.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.53151.log.gz 2024-09-11T09:16:44.680 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.67138.log 2024-09-11T09:16:44.681 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.47739.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.47739.log.gz 2024-09-11T09:16:44.696 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46264.log 2024-09-11T09:16:44.697 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.67138.log: 0.0% -- replaced with /var/log/ceph/cluster2-client.admin.67138.log.gz 2024-09-11T09:16:44.697 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43212.log.gz 2024-09-11T09:16:44.698 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43308.log 2024-09-11T09:16:44.699 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46264.log: 87.9% -- replaced with /var/log/ceph/cluster1-client.mirror.46264.log.gz 2024-09-11T09:16:44.699 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.45605.log 2024-09-11T09:16:44.712 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43308.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.admin.31358.log 2024-09-11T09:16:44.713 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.45605.log: 87.8% -- replaced with /var/log/ceph/cluster1-client.mirror.45605.log.gz 2024-09-11T09:16:44.713 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56876.log 2024-09-11T09:16:44.714 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.admin.31358.log: 80.1% -- replaced with /var/log/ceph/cluster2-client.admin.31358.log.gz 2024-09-11T09:16:44.728 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60802.log 2024-09-11T09:16:44.729 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56876.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.56876.log.gz 2024-09-11T09:16:44.730 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60532.log 2024-09-11T09:16:44.730 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60802.log: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60802.log.gz 2024-09-11T09:16:44.739 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.36061.log.gz 2024-09-11T09:16:44.744 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49324.log 2024-09-11T09:16:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60532.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.60532.log.gz 2024-09-11T09:16:44.745 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63317.log 2024-09-11T09:16:44.746 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49324.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49324.log.gz 2024-09-11T09:16:44.747 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.63704.log 2024-09-11T09:16:44.748 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63317.log: 88.5% -- replaced with /var/log/ceph/cluster2-client.mirror.63317.log.gz 2024-09-11T09:16:44.748 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44104.log 2024-09-11T09:16:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.63704.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.63704.log.gz 2024-09-11T09:16:44.749 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.37429.log 2024-09-11T09:16:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44104.log: 1.3% -- replaced with /var/log/ceph/cluster2-client.mirror.44104.log.gz 2024-09-11T09:16:44.750 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50948.log 2024-09-11T09:16:44.756 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.37429.log: 93.6% -- replaced with /var/log/ceph/cluster2-client.mirror.37429.log.gz 2024-09-11T09:16:44.756 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.56959.log 2024-09-11T09:16:44.758 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50948.log: 92.3% -- replaced with /var/log/ceph/cluster1-client.mirror.50948.log.gz 2024-09-11T09:16:44.759 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.63395.log 2024-09-11T09:16:44.760 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.56959.log: 88.3% -- replaced with /var/log/ceph/cluster2-client.mirror.56959.log.gz 2024-09-11T09:16:44.760 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57785.log 2024-09-11T09:16:44.761 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.63395.log: 87.6% -- replaced with /var/log/ceph/cluster2-client.mirror.63395.log.gz 2024-09-11T09:16:44.761 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.30823.log 2024-09-11T09:16:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57785.log: 90.6% -- replaced with /var/log/ceph/cluster2-client.mirror.57785.log.gz 2024-09-11T09:16:44.764 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54452.log 2024-09-11T09:16:44.776 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.1.30823.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43883.log 2024-09-11T09:16:44.778 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.54452.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.54452.log.gz 2024-09-11T09:16:44.792 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38117.log 2024-09-11T09:16:44.800 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43883.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65587.log 2024-09-11T09:16:44.808 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38117.log: 93.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.58711.log 2024-09-11T09:16:44.809 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65587.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65587.log.gz 2024-09-11T09:16:44.816 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.50233.log 2024-09-11T09:16:44.817 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.58711.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.58711.log.gz 2024-09-11T09:16:44.824 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.38117.log.gz 2024-09-11T09:16:44.824 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53669.log 2024-09-11T09:16:44.832 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.50233.log: 92.4% -- replaced with /var/log/ceph/cluster1-client.mirror.50233.log.gz 2024-09-11T09:16:44.832 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.56064.log 2024-09-11T09:16:44.840 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53669.log: 92.5%gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.35659.log 2024-09-11T09:16:44.842 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.56064.log: 92.0% -- replaced with /var/log/ceph/cluster1-client.mirror.56064.log.gz 2024-09-11T09:16:44.848 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.53669.log.gz 2024-09-11T09:16:44.848 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.31426.log 2024-09-11T09:16:44.855 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.35659.log: 91.7% 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.43308.log.gz 2024-09-11T09:16:44.855 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.35659.log.gz 2024-09-11T09:16:44.865 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39128.log.gz 2024-09-11T09:16:44.868 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.44408.log 2024-09-11T09:16:44.869 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.31426.log: gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57413.log 2024-09-11T09:16:44.869 INFO:teuthology.orchestra.run.smithi077.stderr: 2.3% -- replaced with /var/log/ceph/cluster2-client.mirror.31426.log.gz 2024-09-11T09:16:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.44408.log: 89.8% -- replaced with /var/log/ceph/cluster2-client.mirror.44408.log.gz 2024-09-11T09:16:44.870 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42301.log 2024-09-11T09:16:44.873 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57413.log: 90.5% -- replaced with /var/log/ceph/cluster2-client.mirror.57413.log.gz 2024-09-11T09:16:44.873 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.34406.log 2024-09-11T09:16:44.874 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42301.log: 79.5% -- replaced with /var/log/ceph/cluster2-client.mirror.42301.log.gz 2024-09-11T09:16:44.874 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.33430.log 2024-09-11T09:16:44.875 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.34406.log: 89.9% -- replaced with /var/log/ceph/cluster2-client.mirror.34406.log.gz 2024-09-11T09:16:44.876 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.1.49422.log 2024-09-11T09:16:44.877 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.33430.log: 82.1% -- replaced with /var/log/ceph/cluster2-client.mirror.33430.log.gz 2024-09-11T09:16:44.877 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.59269.log 2024-09-11T09:16:44.892 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.1.49422.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44706.log 2024-09-11T09:16:44.893 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.59269.log: 88.1% -- replaced with /var/log/ceph/cluster2-client.mirror.59269.log.gz 2024-09-11T09:16:44.912 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-osd.2.log 2024-09-11T09:16:44.914 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44706.log: 92.1% -- replaced with /var/log/ceph/cluster1-client.mirror.44706.log.gz 2024-09-11T09:16:44.919 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster2-client.mirror.42597.log.gz 2024-09-11T09:16:44.919 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40525.log 2024-09-11T09:16:44.925 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-osd.2.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.39769.log.gz 2024-09-11T09:16:44.936 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.65081.log 2024-09-11T09:16:44.937 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.40525.log: 87.8% -- replaced with /var/log/ceph/cluster2-client.mirror.40525.log.gz 2024-09-11T09:16:44.937 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57393.log 2024-09-11T09:16:44.938 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.65081.log: 86.9% -- replaced with /var/log/ceph/cluster1-client.mirror.65081.log.gz 2024-09-11T09:16:44.939 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.3.41876.log 2024-09-11T09:16:44.940 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57393.log: 88.4% -- replaced with /var/log/ceph/cluster2-client.mirror.57393.log.gz 2024-09-11T09:16:44.940 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.42380.log 2024-09-11T09:16:44.941 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.3.41876.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.1.49422.log.gz 2024-09-11T09:16:44.956 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.52176.log 2024-09-11T09:16:44.957 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.42380.log: 79.9% -- replaced with /var/log/ceph/cluster2-client.mirror.42380.log.gz 2024-09-11T09:16:44.957 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.admin.30621.log 2024-09-11T09:16:44.960 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.52176.log: 92.6% -- replaced with /var/log/ceph/cluster2-client.mirror.52176.log.gz 2024-09-11T09:16:44.960 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.32766.log 2024-09-11T09:16:44.961 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.admin.30621.log: 0.0% -- replaced with /var/log/ceph/cluster1-client.admin.30621.log.gz 2024-09-11T09:16:44.961 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.59835.log 2024-09-11T09:16:44.962 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.32766.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.32766.log.gz 2024-09-11T09:16:44.962 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.57992.log 2024-09-11T09:16:44.963 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.59835.log: 86.8% -- replaced with /var/log/ceph/cluster1-client.mirror.59835.log.gz 2024-09-11T09:16:44.963 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.37321.log 2024-09-11T09:16:44.964 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.57992.log: 88.2% -- replaced with /var/log/ceph/cluster2-client.mirror.57992.log.gz 2024-09-11T09:16:44.965 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.43286.log 2024-09-11T09:16:44.966 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.37321.log: 88.6% -- replaced with /var/log/ceph/cluster1-client.mirror.37321.log.gz 2024-09-11T09:16:44.966 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.49927.log 2024-09-11T09:16:44.967 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.43286.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.43286.log.gz 2024-09-11T09:16:44.967 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.46023.log 2024-09-11T09:16:44.968 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.49927.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.49927.log.gz 2024-09-11T09:16:44.969 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41639.log 2024-09-11T09:16:44.970 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.46023.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.46023.log.gz 2024-09-11T09:16:44.970 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55610.log 2024-09-11T09:16:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41639.log: 87.5% -- replaced with /var/log/ceph/cluster2-client.mirror.41639.log.gz 2024-09-11T09:16:44.971 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.53777.log 2024-09-11T09:16:44.973 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55610.log: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.55610.log.gz 2024-09-11T09:16:44.974 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.38624.log 2024-09-11T09:16:44.975 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.53777.log: 87.7% -- replaced with /var/log/ceph/cluster2-client.mirror.53777.log.gz 2024-09-11T09:16:44.975 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.38051.log 2024-09-11T09:16:44.978 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.38624.log: 91.1% -- replaced with /var/log/ceph/cluster2-client.mirror.38624.log.gz 2024-09-11T09:16:44.978 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.41249.log 2024-09-11T09:16:44.979 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.38051.log: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.43883.log.gz 2024-09-11T09:16:44.982 INFO:teuthology.orchestra.run.smithi077.stderr: 93.8% -- replaced with /var/log/ceph/cluster1-client.mirror.1.30823.log.gz 2024-09-11T09:16:44.984 INFO:teuthology.orchestra.run.smithi077.stderr: 93.5% -- replaced with /var/log/ceph/cluster1-client.mirror.38051.log.gz 2024-09-11T09:16:44.984 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55774.log 2024-09-11T09:16:44.984 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.40767.log 2024-09-11T09:16:44.985 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster2-client.mirror.41249.log: /var/log/ceph/cluster1-client.mirror.55774.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.60420.log 2024-09-11T09:16:44.985 INFO:teuthology.orchestra.run.smithi077.stderr: 87.7%/var/log/ceph/cluster2-client.mirror.40767.log: -- replaced with /var/log/ceph/cluster1-client.mirror.55774.log.gz 2024-09-11T09:16:44.986 INFO:teuthology.orchestra.run.smithi077.stderr: 87.5%gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.55304.log 2024-09-11T09:16:44.986 INFO:teuthology.orchestra.run.smithi077.stderr: -- replaced with /var/log/ceph/cluster2-client.mirror.40767.log.gz 2024-09-11T09:16:44.986 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.60420.log: gzip -5 --verbose -- /var/log/ceph/cluster1-client.mirror.44894.log 2024-09-11T09:16:44.987 INFO:teuthology.orchestra.run.smithi077.stderr: 86.7% -- replaced with /var/log/ceph/cluster1-client.mirror.60420.log.gz 2024-09-11T09:16:44.987 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.55304.log: 87.3% -- replaced with /var/log/ceph/cluster1-client.mirror.55304.log.gz 2024-09-11T09:16:44.987 INFO:teuthology.orchestra.run.smithi077.stderr:gzip -5 --verbose -- /var/log/ceph/cluster2-client.mirror.54372.log 2024-09-11T09:16:44.988 INFO:teuthology.orchestra.run.smithi077.stderr:/var/log/ceph/cluster1-client.mirror.44894.log: /var/log/ceph/cluster2-client.mirror.54372.log: 87.7% -- replaced with /var/log/ceph/cluster1-client.mirror.44894.log.gz 2024-09-11T09:16:44.989 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster2-client.mirror.54372.log.gz 2024-09-11T09:16:45.056 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.32256.log.gz 2024-09-11T09:16:45.137 INFO:teuthology.orchestra.run.smithi077.stderr: 92.5% -- replaced with /var/log/ceph/cluster1-client.mirror.3.41876.log.gz 2024-09-11T09:16:45.204 INFO:teuthology.orchestra.run.smithi077.stderr: 93.9% -- replaced with /var/log/ceph/cluster2-client.mirror.41249.log.gz 2024-09-11T09:16:48.819 INFO:teuthology.orchestra.run.smithi073.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.2.log.gz 2024-09-11T09:16:48.859 INFO:teuthology.orchestra.run.smithi073.stderr: 94.3% -- replaced with /var/log/ceph/cluster1-osd.1.log.gz 2024-09-11T09:16:49.571 INFO:teuthology.orchestra.run.smithi073.stderr: 94.4% -- replaced with /var/log/ceph/cluster1-osd.0.log.gz 2024-09-11T09:16:49.573 INFO:teuthology.orchestra.run.smithi073.stderr: 2024-09-11T09:16:49.573 INFO:teuthology.orchestra.run.smithi073.stderr:real 0m8.452s 2024-09-11T09:16:49.573 INFO:teuthology.orchestra.run.smithi073.stderr:user 0m24.325s 2024-09-11T09:16:49.573 INFO:teuthology.orchestra.run.smithi073.stderr:sys 0m0.960s 2024-09-11T09:17:25.036 INFO:teuthology.orchestra.run.smithi077.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.1.log.gz 2024-09-11T09:17:28.246 INFO:teuthology.orchestra.run.smithi077.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.2.log.gz 2024-09-11T09:17:39.491 INFO:teuthology.orchestra.run.smithi077.stderr: 93.6% -- replaced with /var/log/ceph/cluster2-osd.0.log.gz 2024-09-11T09:17:39.493 INFO:teuthology.orchestra.run.smithi077.stderr: 2024-09-11T09:17:39.493 INFO:teuthology.orchestra.run.smithi077.stderr:real 0m58.361s 2024-09-11T09:17:39.493 INFO:teuthology.orchestra.run.smithi077.stderr:user 2m41.717s 2024-09-11T09:17:39.493 INFO:teuthology.orchestra.run.smithi077.stderr:sys 0m6.319s 2024-09-11T09:17:39.494 INFO:tasks.ceph:Archiving logs... 2024-09-11T09:17:39.494 DEBUG:teuthology.misc:Transferring archived files from smithi073:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/remote/smithi073/log 2024-09-11T09:17:39.495 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T09:17:42.310 DEBUG:teuthology.misc:Transferring archived files from smithi077:/var/log/ceph to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/remote/smithi077/log 2024-09-11T09:17:42.311 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /var/log/ceph -- . 2024-09-11T09:18:02.521 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-09-11T09:18:02.535 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-09-11T09:18:02.536 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T09:18:02.538 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-09-11T09:18:02.598 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-09-11T09:18:02.619 INFO:teuthology.task.clock:Checking final clock skew... 2024-09-11T09:18:02.619 DEBUG:teuthology.orchestra.run.smithi073:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:18:02.622 DEBUG:teuthology.orchestra.run.smithi077:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-09-11T09:18:02.632 INFO:teuthology.orchestra.run.smithi073.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T09:18:02.633 INFO:teuthology.orchestra.run.smithi073.stdout:============================================================================== 2024-09-11T09:18:02.633 INFO:teuthology.orchestra.run.smithi073.stdout:+hv01.front.sepi 67.205.162.81 3 u 63 64 377 0.091 -1.125 0.509 2024-09-11T09:18:02.633 INFO:teuthology.orchestra.run.smithi073.stdout:+hv02.front.sepi 63.231.80.2 3 u - 64 377 0.089 -5.196 0.618 2024-09-11T09:18:02.633 INFO:teuthology.orchestra.run.smithi073.stdout:*hv03.front.sepi 74.6.168.72 3 u 5 64 377 0.093 -4.335 0.857 2024-09-11T09:18:02.633 INFO:teuthology.orchestra.run.smithi073.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-11T09:18:02.636 INFO:teuthology.orchestra.run.smithi077.stdout: remote refid st t when poll reach delay offset jitter 2024-09-11T09:18:02.636 INFO:teuthology.orchestra.run.smithi077.stdout:============================================================================== 2024-09-11T09:18:02.636 INFO:teuthology.orchestra.run.smithi077.stdout:+hv01.front.sepi 67.205.162.81 3 u 2 64 377 0.056 +0.005 0.852 2024-09-11T09:18:02.636 INFO:teuthology.orchestra.run.smithi077.stdout:+hv02.front.sepi 63.231.80.2 3 u - 64 377 0.098 -5.631 0.706 2024-09-11T09:18:02.636 INFO:teuthology.orchestra.run.smithi077.stdout:*hv03.front.sepi 74.6.168.72 3 u 3 64 377 0.099 -4.907 0.494 2024-09-11T09:18:02.636 INFO:teuthology.orchestra.run.smithi077.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 +0.000 0.000 2024-09-11T09:18:02.637 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-09-11T09:18:02.732 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-09-11T09:18:02.733 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-09-11T09:18:02.802 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-09-11T09:18:02.840 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-09-11T09:18:02.911 INFO:teuthology.task.internal:Duration was 2951.291754 seconds 2024-09-11T09:18:02.911 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-09-11T09:18:03.001 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-09-11T09:18:03.002 DEBUG:teuthology.orchestra.run.smithi073:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T09:18:03.004 DEBUG:teuthology.orchestra.run.smithi077:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-09-11T09:18:03.042 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-09-11T09:18:03.042 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi073.front.sepia.ceph.com 2024-09-11T09:18:03.043 DEBUG:teuthology.orchestra.run.smithi073:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T09:18:03.093 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi077.front.sepia.ceph.com 2024-09-11T09:18:03.093 DEBUG:teuthology.orchestra.run.smithi077:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-09-11T09:18:03.103 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-09-11T09:18:03.104 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T09:18:03.137 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-09-11T09:18:03.174 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-09-11T09:18:03.174 DEBUG:teuthology.orchestra.run.smithi073:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T09:18:03.205 DEBUG:teuthology.orchestra.run.smithi077:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-09-11T09:18:03.273 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-09-11T09:18:03.286 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-09-11T09:18:03.286 DEBUG:teuthology.orchestra.run.smithi073:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T09:18:03.298 DEBUG:teuthology.orchestra.run.smithi077:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-09-11T09:18:03.309 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-09-11T09:18:03.320 DEBUG:teuthology.orchestra.run.smithi073:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:18:03.341 DEBUG:teuthology.orchestra.run.smithi077:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:18:03.350 INFO:teuthology.orchestra.run.smithi073.stdout:kernel.core_pattern = core 2024-09-11T09:18:03.361 INFO:teuthology.orchestra.run.smithi077.stdout:kernel.core_pattern = core 2024-09-11T09:18:03.457 DEBUG:teuthology.orchestra.run.smithi073:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:18:03.503 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:18:03.504 DEBUG:teuthology.orchestra.run.smithi077:> test -e /home/ubuntu/cephtest/archive/coredump 2024-09-11T09:18:03.509 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-09-11T09:18:03.509 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-09-11T09:18:03.522 INFO:teuthology.task.internal:Transferring archived files... 2024-09-11T09:18:03.522 DEBUG:teuthology.misc:Transferring archived files from smithi073:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/remote/smithi073 2024-09-11T09:18:03.522 DEBUG:teuthology.orchestra.run.smithi073:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T09:18:03.569 DEBUG:teuthology.misc:Transferring archived files from smithi077:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/teuthology-2024-09-10_21:16:03-rbd-squid-distro-default-smithi/7900659/remote/smithi077 2024-09-11T09:18:03.569 DEBUG:teuthology.orchestra.run.smithi077:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-09-11T09:18:03.590 INFO:teuthology.task.internal:Removing archive directory... 2024-09-11T09:18:03.591 DEBUG:teuthology.orchestra.run.smithi073:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T09:18:03.609 DEBUG:teuthology.orchestra.run.smithi077:> rm -rf -- /home/ubuntu/cephtest/archive 2024-09-11T09:18:03.637 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-09-11T09:18:03.649 INFO:teuthology.task.internal:Not uploading archives. 2024-09-11T09:18:03.649 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-09-11T09:18:03.778 INFO:teuthology.task.internal:Tidying up after the test... 2024-09-11T09:18:03.778 DEBUG:teuthology.orchestra.run.smithi073:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T09:18:03.781 DEBUG:teuthology.orchestra.run.smithi077:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-09-11T09:18:03.784 INFO:teuthology.orchestra.run.smithi073.stdout: 658462 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 09:18 /home/ubuntu/cephtest 2024-09-11T09:18:03.787 INFO:teuthology.orchestra.run.smithi077.stdout: 658640 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Sep 11 09:18 /home/ubuntu/cephtest 2024-09-11T09:18:03.789 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-09-11T09:18:03.851 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-09-11T09:18:03.966 INFO:teuthology.run:Summary data: description: rbd/mirror-thrash/{base/install clients/mirror cluster/{2-node openstack} conf/{disable-pool-app} msgr-failures/few objectstore/bluestore-low-osd-mem-target policy/simple rbd-mirror/four-per-cluster supported-random-distro$/{ubuntu_latest} workloads/rbd-mirror-snapshot-stress-workunit-exclusive-lock} duration: 2951.291754245758 flavor: default owner: scheduled_teuthology@teuthology success: true 2024-09-11T09:18:03.966 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-09-11T09:18:04.047 INFO:teuthology.run:pass